KR102142183B1 - Method for estimating emotion based on psychological activity and biosignal of user and system therefor - Google Patents

Abstract

Disclosed is a method and system for estimating emotions based on a user's psychological activity and bio signals. The emotion estimation method according to an embodiment of the present invention is a motion component, a motivation component, and an adaptation component through learning a user's emotional situation using an image sequence. ); Modeling a balance model and an arousal model based on the calculated motion component, synchronous component and adaptive component; And generating an emotional curve based on the emotional situation expressed on the emotional space by the modeled balance model and auraun model.

Description

Method and estimating emotion based on psychological activity and biosignal of user and system therefor}

The present invention relates to a technique for inferring a user's emotions, and more particularly, to a method and a system for estimating a user's emotions based on the user's psychological activity and bio signals.

Emotional awareness is gaining much attention recently because it facilitates understanding of the emotional interactions between people and computers by measuring emotional states such as joy, excitement and fear. To model a person's emotional state, subjective self-reports that describe in detail the feelings a person feels can provide useful information that can be used to display (or label) his emotional state. For example, the emotion-assessment tool, Self-Assessment Manikin (SAM), uses a graphic scale that depicts a cartoon character expressing three emotional elements: pleasure, arousal, and dominance. In most cases, the participant's self-assessment is performed in a laboratory environment in which the participant sits in a comfortable chair and displays the evaluation level. This method of evaluation has problems related to validity and confirmation, for example, a participant cannot accurately answer the feelings he or she feels and instead of answering an answer that others think will be answered. In addition, this method of evaluation can only produce immediate human emotions for complex emotional conditions, and only provides a limited understanding of the emotional dynamics in everyday life. Therefore, it is important to provide an automatic method for displaying human emotions caused in real situations.

However, little research has been done on affective labeling of real situations. It is difficult to quantify an emotional response based on an understanding of a situation. It requires a cognitive understanding of the actual objects that humans interact with, which can determine the emotional level of expected emotions based on the interactions. The most difficult problems are quantifying the level of subject-dependent emotions, finding visual attention in the human cortex, and reducing artifacts in the real world.

Providing emotional labels can improve specialty areas such as content recommendation systems, and quantify the subject's emotional response to stimuli that are the cause of the emotional dimension using two methods: explicit tagging and implicit tagging.

Human emotion can be conceptualized in three important ways: Valence (V), Arousal (A), and Dominance (D). Balance is a type of emotion and characterizes an emotional state or reaction, from unpleasant or negative emotions to pleasant, happy and positive emotions. Arousal is the intensity of emotions, which characterizes an emotional state or reaction from drowsiness or boredom to extreme excitement. Dominance distinguishes emotional states with valence and arousal from "no control" to "full control". For example, sadness and anger have similar valance and arousal values, but are distinguished from dominance values. The full range of human emotions can be expressed as a series of points in a 3D VAC coordinate space. Conversely, each basic emotion can be expressed as a bipolar entity, and features all emotions due to valence and arousal as shown in FIG. 1A, and the emotion label can be expressed at various locations on the 2D VA plane.

In addition, according to various conventional studies, it can be seen that the emotional response mapped to the emotional coordinate system is approximately a parabola as illustrated in FIG. 1B. In one example prior art, a parabolic surface was used to define the character of an agent by assigning temperament, mood, and emotion to an emotional agent.

Explicit approaches provide labels by requiring users to report their feelings in response to a given event or stimulus. The International Affective Picture System (IAPS) is a common data set used to obtain emotional labels using explicit self-reporting tools such as SAM. As shown in Figure 2, SAM measures a wide range of emotional responses to various stimuli associated with valence (from positive to negative), arousal (from high to low) and dominance (from low to high). It is a picture-based evaluation technique. Emotional labels obtained from explicit self-reporting tools are considered actual data on emotional state and are used to build a reliable emotional recognition system.

However, the main disadvantage of explicit approaches to expressing human emotions is the intrusiveness of the reporting process. Conversely, the implicit emotional labeling approach is a non-intervention measure. Labeling can be achieved by exposing the user to the stimulus and recording the response to the exposure. Visual and motion-related functions are important elements in emotion recognition. The technique of one conventional embodiment used the face change characteristic to label human emotions, and the technique of another conventional example studied the target motion as a visual characteristic that responds to human emotion, and emotional arousal when the intensity of motion increases It showed that the level could increase. The technique of another conventional embodiment has developed a method of characterizing arousal using motion intensity and a shot change rate, and the technique of another conventional embodiment uses motion activity to determine the arousal level and continuous arousal Changes are expressed as curves.

Another method is to extract emotional features of psychological behaviors and associate them with emotional states, and many studies have proposed methods to measure emotional behaviors. For example, arm movements such as flexion and extension have been used to investigate positive and negative interactions between responses to approach and avoidance behaviors. The technique of one conventional embodiment used a joystick to determine whether positive and negative stimuli promote approach and withdrawal behavior. Participants were instructed to control the joystick by pulling the joystick to increase the size of the stimulus or by pushing the joystick to reduce the size of the stimulus, and their metrics can distinguish adolescents' approach and avoidance behaviors that respond to positive or negative stimuli. However, the technology is limited to a controlled experimental setup, and uses some limited perceptual tasks that require the use of specific equipment and that the participant does not interact with the system in real time.

Embodiments of the present invention provide a method and a system for estimating a user's emotion based on a user's psychological activity and a biosignal.

Embodiments of the present invention provide a method and a system capable of extracting an emotion value felt by a user based on a user's behavior.

The emotion estimation method according to an embodiment of the present invention is a motion component, a motivation component, and an adaptation component through learning a user's emotional situation using an image sequence. ); Modeling a balance model and an arousal model based on the calculated motion component, synchronous component and adaptive component; And generating an emotional curve based on the emotional situation expressed on the emotional space by the modeled balance model and auraun model.

The calculating step may calculate the motion component by characterizing and quantifying motion of an emotional object between adjacent frames using optical flow estimation for the image sequence.

The calculating step may calculate the motion component by removing motion noise using motion activity estimated in each image frame of the image sequence and accelerometer data collected by an accelerometer attached to the user's body.

The calculating step may obtain a protruding area corresponding to a user's interest in an image frame of the image sequence, and calculate the synchronous component based on the obtained protruding area.

The calculating step may calculate the synchronous component by calculating divergence and rotation using an optical flow of a predetermined area around the object of interest in the image frame of the image sequence.

The calculating step may calculate the adaptive component based on the emotional situation length accumulated up to the current frame of the image sequence that changes over time.

The modeling step may model the arousal model using the calculated motion component and adaptive component.

In the modeling step, the balance model may be modeled using the aurajeol model and the synchronous component.

An emotion estimation method according to another embodiment of the present invention includes modeling a balance model and an arousal model based on a user's behavior using an image sequence; Generating an emotional curve based on the emotional situation expressed on the emotional space by the modeled balance model and auraun model; And estimating the emotion value of the user using the emotion curve.

The modeling step includes calculating a motion component, a motivation component, and an adaptation component based on the user's behavior; And modeling a balance model and an arousal model based on the calculated motion component, synchronous component, and adaptive component.

The modeling step may detect the user's interest from the image sequence, and model the balance model and the arousal model based on the interaction with the sensed interest.

The emotion estimation system according to an embodiment of the present invention is a motion component, a motivation component, and an adaptation component through learning a user's emotional situation using an image sequence. ) Calculating unit; A modeling unit that models a balance model and an arousal model based on the calculated motion component, synchronous component, and adaptive component; And a generating unit generating an emotional curve based on the emotional situation expressed on the emotional space by the modeled balance model and the aurajeol model.

An emotion estimation system according to another embodiment of the present invention includes a modeling unit for modeling a balance model and an arousal model based on a user's behavior using an image sequence; A generation unit generating an emotion curve based on the emotional situation expressed on the emotional space by the modeled balance model and the auraun model; And an estimation unit for estimating the emotion value of the user using the emotional curve.

According to embodiments of the present invention, it is possible to provide a calculation framework for providing emotional labels in an actual situation.

Existing emotion recognition systems allow users to passively calculate their emotions by self-assessment self-assessment and recognize emotions accordingly, while the present invention provides emotions from user's behavior without such self-assessment. The value can be calculated automatically.

In addition, since the present invention infers emotion values through actions performed by users in real life, automation of an existing emotion recognition system can be achieved and can be applied to various applications in a real life environment.

The present invention can be applied to a medical field through application of an early diagnosis technique for depression, a technique for grasping a user's emotional response to content in virtual/augmented reality, and the like.

In addition, the present invention can be expanded to the wearable market according to the development of miniaturization technology of sensors for measuring brain waves, and can also be used in daily life through the manufacture of brain waves measuring wearable devices that can be linked to a user's smartphone.

FIG. 1 shows an exemplary diagram for the distribution of emotions and the shape of a parabola in the balance-arrow space.
Figure 2 shows an exemplary diagram for explaining a self-assessment manikin (SAM).
3 is a conceptual diagram for a framework according to an embodiment of the present invention.
4 shows an exemplary diagram for describing a synchronous component.
FIG. 5 shows the results of a 5x cross-effectiveness scheme for balance and aura-class distance between SAM and the framework of the present invention for different parameters.
FIG. 6 is a diagram illustrating an example of an emotional curve that combines an arousal curve and a balance curve.
FIG. 7 shows an exemplary diagram for the results of the classification procedure through the 5-fold cross-effectiveness scheme for all participants.

Advantages and features of the present invention, and methods for achieving them will be clarified with reference to embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in various different forms, only the embodiments allow the disclosure of the present invention to be complete, and common knowledge in the art to which the present invention pertains. It is provided to fully inform the person having the scope of the invention, and the present invention is only defined by the scope of the claims.

The terminology used herein is for describing the embodiments, and is not intended to limit the present invention. In the present specification, the singular form also includes the plural form unless otherwise specified in the phrase. As used herein, “comprises” and/or “comprising” refers to the components, steps, operations, and/or elements mentioned above of one or more other components, steps, operations, and/or elements. Presence or addition is not excluded.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings commonly understood by those skilled in the art to which the present invention pertains. In addition, terms that are defined in advance are generally not to be interpreted ideally or excessively unless specifically defined.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the accompanying drawings. The same reference numerals are used for the same components in the drawings, and duplicate descriptions for the same components are omitted.

Embodiments of the present invention provide a low-level function-based framework for modeling emotional video content, such as a two-dimensional emotional coordinate space, for example, a valence-arousal emotional coordinate space The main goal is to provide a framework for expressing and modeling emotional situations with a series of points.

Here, the present invention can detect the visual interest in the human cortex, which is the basis of psychological behavior, and quantify the expected emotion based on the interaction with the object of interest.

The present invention can express the result of the spatiotemporal situation as a polynomial curve called an emotional curve.

Here, the video can be expressed as an emotional curve stably depicting a transition expected from one feeling to another while watching a video, and the emotional curve is an International Affective Picture System (IAPS) (PJ Lang, MM Bradley, and BN Cuthbert, "International affective picture system (iaps): Technical manual and affective ratings," NIMH Center for the Study of Emotion and Attention, pp. 39-58, 1997.) and International Affective Digitized Sounds system (IADS) (M Bradley and P. Lang, "International affective digitized sounds (iads): Technical manual and affective ratings," Center for Research in Psychophysiology, Univ. Florida, Gainesville, FL, 1991.) Can be based.

IAPS and IADS include measurements of emotional responses to calibrated audiovisual stimuli, and while both systems aim to collect a wide range of emotions from audiovisual content, the fact that the emotional responses mapped to the emotional coordinate system is roughly parabolic It has been determined through several studies. For example, a parabolic surface can be used to define an agent's personality by assigning temperament, mood, and emotion to an emotional agent.

The present invention derives a model that derives an emotional curve from low-level features extracted from an image sequence, similar to the above, the model in the present invention reflects the emotional behavior of humans based on events of life and affects emotional situations Approach-withdrawal theory is used to calculate the emotional curve representing.

In the framework of the present invention, emotional situations are referred to as'emotional curves'. Here, the emotional curve may mean a polynomial curve fitted to a set of points in the valence-arousal emotional space. In addition, in order to model and express an emotional situation in a real environment, environmental information may be collected using an easily wearable device so that a user can freely act in a daily situation. The device can consist of a front camera, an accelerometer and a small physiological sensor. The present invention provides appropriate emotional labels for learning emotional context expressions using data collected from these devices and for knowing physiological changes that can be used to classify human emotions.

Emotional situations in the present invention can be defined as a specific arrangement of emotional entities, and the present invention provides a framework for modeling and expressing such emotional situations. Here, emotional reality can be a variety of real-world objects where people meet, interact, and feel emotional reactions.

When the emotional situation is modeled by the present invention, it is possible to understand the content of human interaction, and by expressing this situation, it is possible to determine the expected level of emotion based on the interaction. Therefore, the framework of the present invention can help to reduce the semantic difference between cognitive emotions and emotional emotions in real situations.

The present invention will be described in detail as follows.

3 illustrates a framework according to an embodiment of the present invention, the framework of the present invention provides emotional labeling of an image sequence in an actual scenario.

The framework of the present invention learns and expresses an emotional situation based on an egocentric image in order to quantify the feeling of behavior tendency from the emotional behavior faced by emotional content in a situation. The present invention can take a self-centered image every time frame, use auxiliary accelerometer data as input, and output two emotion points in a valence-arousal space.

Here, the learned point can be expressed as a polynomial curve called an affective curve.

As shown in FIG. 3, the framework of the present invention expresses an emotional situation using three components calculated after calculating a motion component, a synchronous component, and an adaptive component from an image sequence, and a two-dimensional emotional space The emotional curve is generated using the emotional situation expressed on the image, and emotional labeling is provided using the generated emotional curve.

Emotional situation learning

One conventional technology (A. Hanjalic and L.-Q. Xu, "Affective video content representation and modeling," IEEE transactions on multimedia, vol. 7, no. 1, pp. 143-154, 2005.) Emotion curves for valence and arousal spaces were developed based on three criteria used to justify psychologically. The present invention modifies the above-mentioned criteria in order to apply the framework of the present invention to the actual situation.

(1) Comparability: The values of valence, arousal, and resulting emotional curves obtained in different situations can be compared for similar types of emotional behavior. This criterion naturally imposes standardization and scaling requirements when calculating the time curve.

(2) Compatibility: The shape of the emotional curve reflects the situation when a specific time is given in the valence-arousal emotional space. When the situation ends, the shape of the curve becomes an approximate parabolic shape in the 2D emotional space.

(3) Smoothness: indicates the degree of emotion retention in the previous frame. This ensures that the emotional proportion of content involved in eliciting human emotions does not suddenly change between successive frames of the situation.

1) Emotional characteristics of emotional situation learning

The present invention can observe various emotional expressions used in situations. The present invention uses general functions A(k) and V(k) for arousal and valence in frame k to model these emotional phenomena as arousal and valence. The two functions may have a suitable form of function that integrates the three components of the motion component of the emotional response, the motivation component of the emotional approach-avoidance behavior, and the adaptive component of the emotional situation.

a) Motion component of emotional response

Several previous studies have reported that motion is one of the most important visual features affecting an individual's emotional response. Motion is expressive and can evoke strong emotional reactions. Studies on the effect of the subject's motion on the human emotional response indicate that arousal increases as the motion intensity increases.

The present invention calculates the motion component to estimate the motion activity m(k) in each video frame k.

의 평균 크기는 모션 활동성을 공식화하는데 사용될 수 있다.Here, the present invention can characterize and quantify the motion of an emotional object between adjacent frames using optical flow estimation, and all estimated motion vectors

The average size of can be used to formulate motion activity.

The motion activity m(k) can be expressed as <Equation 1> below.

[Equation 1]

는 프레임 k에서의 모션 벡터 i를 의미하고, B는 프레임 k에서의 모션 벡터의 수를 의미할 수 있다.here,

Denotes a motion vector i in frame k, and B denotes the number of motion vectors in frame k.

에 가속도계 데이터

를 사용할 수 있으며, 모션 컴포넌트

는 아래 <수학식 2>와 같이 나타낼 수 있다.Since the self-centered image is usually captured by a camera attached to the human body, the image may contain motion noise that contaminates the motion activity m(k). In order to remove such motion noise, the present invention is a motion component

Accelerometer data

You can use motion components

Can be expressed as <Equation 2> below.

[Equation 2]

는 0과 1 사이에서 정규화된 가우시안 평활 결과를 의미할 수 있다.here,

Can mean a Gaussian smoothing result normalized between 0 and 1.

는 모션 잡음이 증가하면 arousal이 감소한다는 것을 의미한다.Because these are not real factors of motion activity

Means that the arousal decreases as the motion noise increases.

FIG. 4 shows a conceptual diagram of motion components, as shown in FIG. 4, visual attention (or interest) from an image in each video frame k is detected in the human cortex by an extruded map. By generating a binary protrusion map using the parabolic protrusion map detected from the image and generating a synchronization map using the generated binary protrusion map, the emotional access-avoidance behavior related to the object of interest can be calculated using the synchronization map. . That is, in the area covered by the binary derivation map, the emotional approach-avoidance behavior related to the object of interest can be calculated using the optical flow around the object.

b) Emotional approach--motivation component of avoidance behavior

를 계산한다. 동기 구성 요소의 계산은 먼저 시각적인 주의(visual attention)을 통해 참가자의 의도를 파악하여 수행된다.The present invention is a synchronous component for modeling emotional behavior occurring in response to emotional content in various situations similar to the motion component

To calculate. The calculation of the motivational component is performed by first grasping the participant's intention through visual attention.

Here, predicting the position of the visual attention maintained at a specific fixed point may be performed with saliency prediction or detection.

In computer vision, robotics, and neuroscience, this is happening and is getting more and more attention. The present invention may use a saliency model, for example, a saliency attentive model, to obtain the most protruding area in the image frame.

Here, the protrusion model is constructed in a scene by constructing a convolutional LSTM having a set of features calculated by gaze priors multi-learned with dilated convolutional networks (DCNs) and an object detector. Human eye fixations can be predicted.

The present invention uses a final protrusion map, which is a probability map having a value in [0, 1], to generate a binary protrusion map having a threshold value t _h . In the binary overhanging map, white becomes the main fixed area of interest (or attention) for the participant. The present invention learns the emotional approach-avoidance behavior related to the object of interest by calculating the divergence and rotation using the optical flow around the object of interest in each video frame k based on the protruding object, which is the prediction result of the human visual attention. This is possible because approaching a single object is the same as enlarging the object, and has the same effect on the divergence of the flow vector surrounding the center point of the extruded object.

At this time, the present invention can calculate the flow using multi-scale block-based matching between adjacent frames.

는 아래 <수학식 3>과 같이 나타낼 수 있다.The present invention describes six primitive optical flow patterns, e.g., rotation around the vertical axis, rotation around the horizontal axis, approaching the target direction, rotation around the optical axis of the image plane, and complex hyperbolic flows to describe synchronized behavior. Standardize flows with hyperbolic flows. Where, given a motion vector, the velocity at pixel c(x, y)

Can be expressed as <Equation 3> below.

[Equation 3]

는 픽셀 c₀에서의 속도를 의미하고,

는 아래 <수학식 4>와 같이 정의되는 행렬일 수 있다.here,

Is the velocity at pixel c ₀ ,

May be a matrix defined as <Equation 4> below.

[Equation 4]

는 아래 <수학식 5>와 같이 분해될 수 있다.At this time, the matrix

Can be decomposed as in <Equation 5> below.

[Equation 5]

와

은 각각 발산 옵티컬 플로우와 회전 옵티컬 플로우를 의미하고,

과

는 상이한 유형의 쌍곡선 옵티컬 플로우를 의미할 수 있다.here,

Wow

Is a divergent optical flow and a rotating optical flow, respectively,

and

Can mean different types of hyperbolic optical flows.

The matrix D, R, H ₁ and H ₂ may be as shown in <Equation 6> below.

[Equation 6]

로 특징지어 질 수 있다. 본 발명은 관심 대상의 옵티컬 플로우 필드가 주어지면 상기 수학식 5와 최소 자승 오차 방법(least squared error method)을 이용하여 파라미터 벡터를 추정한다. 파라미터 u는 오른쪽 및 왼쪽 회전과 연관되고,

는 위쪽 및 아래쪽 방향과 연관되며, d는 대상 접근 및 회피와 연관되고, 나머지 세 파라미터 r, h₁, h₂는 결합된 모션을 나타낸다.The velocity of the motion vector field is approximately 6 parameters, for example,

Can be characterized as. The present invention estimates a parameter vector using Equation 5 and the least squared error method given the optical flow field of interest. The parameter u is associated with right and left rotation,

Is associated with the upward and downward directions, d is associated with object approach and avoidance, and the remaining three parameters r, h ₁ and h ₂ represent combined motion.

를 계산할 있으며, 아래 <수학식 7>과 같이 나타낼 수 있다.Synchronous component in frame k with 6 parameters

Can be calculated and can be expressed as <Equation 7> below.

[Equation 7]

Here, X and Y may mean the width and height of the optical flow field of interest.

c) the adaptation component of the emotional situation

를 도출함으로써, 상황의 감정적 적응을 모델링할 수 있으며, 함수

는 아래 <수학식 8>과 같이 나타낼 수 있다.The present invention uses context lengths that change over time to represent a link between a user's feelings and an intention to adapt to the situation. The present invention functions at frame k

By deriving, we can model the emotional adaptation of the situation,

Can be expressed as <Equation 8> below.

[Equation 8]

및

은 함수

의 형상을 결정하고,

는 정서적 상황의 길이로, 현재 프레임 k까지의 정서적 상황의 누적 합계를 의미할 수 있다.here,

And

Silver function

Determine the shape of,

Is the length of the emotional situation, and may mean a cumulative sum of emotional situations up to the current frame k.

는 현재 프레임 k의 상황이 이전 프레임 k-1과 동일하면 1만큼 값이 증가한다. 본 발명은 로그 함수 예를 들어, log₁₀ 함수를 사용하여 정서적 상황의 적응 구성 요소를 모델링할 수 있다.function

Increases the value by 1 if the current frame k is the same as the previous frame k-1. The present invention can model an adaptive component of an emotional situation using a logarithmic function, for example a log ₁₀ function.

2) Arousal and Valence models

및 적응 구성요소

의 기여도를 통합하기 위하여 모션 및 적응의 가중된 평균을 사용한다. 상기 함수는 이동 평균 필터를 통해 구성 요소의 이웃하는 로컬 최대 값을 병합하기에 충분히 긴 평활화 윈도우(smoothing window)로 컨벌루션된다. 결과는 0과 1의 범위로 정규화된다. 여기서, 함수 A(k)는 아래 <수학식 9>와 같이 나타낼 수 있다.In the present invention, in order to model arousal, the function A(k) is a motion component along an image sequence in an emotional situation at frame k.

And adaptive components

The weighted average of motion and adaptation is used to incorporate the contribution of. The function is convolved into a smoothing window long enough to merge the neighboring local maximums of the components through a moving average filter. Results are normalized to the range of 0 and 1. Here, the function A(k) can be expressed as <Equation 9> below.

[Equation 9]

는

인 두 함수에 가중치를 부여하기 위한 계수들을 의미할 수 있다.here,

The

It may mean coefficients for weighting the iron function.

The compatibility criterion requires that the emotional curve generated by combining the arousal and valence time curves should contain a region of the valence-arousal coordinate system with a parabolic shape similar to the 2D emotional space. Here, the compatibility criterion may require the arousal value and the absolute value of the associated valence.

Therefore, the range of arousal values can determine the absolute value range of valence.

That is, the present invention can model valence by defining a function r(k) that captures the range dependency of this value in consideration of the value of arousal A(k) in the current frame k.

Here, the function r(k) may be defined as <Equation 10> below, and the valence function V(k) may be expressed as <Equation 11> below.

[Equation 10]

[Equation 11]

는 작은 값이 되고 예상된 감정은 부정적인 경향이 된다.Here, r(k) may mean that the negativity of the expected feeling is determined by the amount of emotional adaptation in the situation. That is, if the participant (subject) wants to avoid the situation,

Becomes a small value and the expected emotion becomes a negative tendency.

에 의해 세밀하게 결정될 수 있으며, 함수 A(k)와 유사하게 함수 V(k)는 동일한 이동 평균 필터로 평활화될 수 있다. 여기서,

는 각각 r(k)와 V(k)의 가중된 평균을 의미할 수 있다.Based on this function, the valence value V(k) is a synchronous component

It can be determined in detail, and similarly to function A(k), function V(k) can be smoothed with the same moving average filter. here,

Can mean the weighted average of r(k) and V(k), respectively.

Expressing emotional situation

The present invention expresses an emotional situation for a 2D emotional space from valence values and arousal values learned by the above-described calculation. The present invention uses two emotion values that fit the Gaussian process regression (GPR) model to approximate the GPR (Gaussian process regression) model. Here, the GPR model may be a non-parametric kernel-based probability model using a linear basic function and an exact fitting method. This can generate an emotional curve as an expression of an emotional trajectory according to a situation sensed by a human being.

device Configuration

The present invention constitutes a wearable device for the framework of the present invention, and allows the user to act freely in everyday situations. The device collects the user's emotions. Here, the device can collect the EEG signal of the user, and capture various effects around the user by using a multi-modal sensor, such as a camera or a physiological sensor.

Emotional situation Dataset

로 정의될 수 있다. 여기서,

이고, Ns는 정서적 상황의 수를 의미하고, Ne는 상황에서 감정적 포인트의 쌍(valence, arousal)의 수를 의미할 수 있다.Emotional situation

Can be defined as here,

, Ns may mean the number of emotional situations, and Ne may mean the number of emotional points (valence, arousal) in the situation.

의 곱에 의하여 결정될 수 있다. 매 30초로부터 모든 감정 포인트들은 정서적 라벨의 단일 집합을 생성하기 위하여 평균될 수 있다. 모든 상황의 지속기간은 세 주석자(annotators)에 의해 매뉴얼적으로 결정될 수 있다. 어떤 상황은 실측 값(ground truth)으로 SAM-등급 라벨을 가질 수 있다. SAM-등급 상황은

로 정의될 수 있으며,

는 Ns 내에서 SAM-등급 상황의 인덱스를 의미할 수 있다. SAM-등급 상황은 본 발명의 프레임워크의 파라미터들을 추정하고 그 성능을 평가하는데 사용된다. 각 상황은 두 채널 EEG 신호와 가속도 측정 데이터를 포함한다. Ne is the frame rate of the camera and the duration of the situation within seconds.

It can be determined by the product of. From every 30 seconds all emotion points can be averaged to create a single set of emotional labels. The duration of any situation can be determined manually by three annotators. Some situations may have a SAM-grade label as the ground truth. SAM-grade situation

Can be defined as

May denote an index of a SAM-grade situation within Ns. The SAM-grade situation is used to estimate the parameters of the framework of the present invention and evaluate its performance. Each situation includes two channel EEG signals and acceleration measurement data.

Parameter setting

의 최소 값을 가지도록 2와 -1로 설정될 수 있다.

와

는 매 오일 동안 최대 값에 의해 결정될 수 있다.To model the emotional situation and express it with an emotional curve, the parameters for each participant are set through a five-fold cross-validation scheme. These parameters have an adaptive component of -1/ in the first frame of the situation.

It can be set to 2 and -1 to have the minimum value of.

Wow

Can be determined by the maximum value for every oil.

,

과

에 대하여 SAM과 본 발명의 프레임워크 간 valence와 arousal 등급 거리에 관한 5배 교차 유효성 스킴 결과를 나타낸 것이다. 정서 커브는 공간과 시간적 상황에서 정서적 역학(affective dynamics)을 표현하고 정서적 라벨의 복수 쌍을 포함하기 때문에 SAM 등급의 쌍과 비교할 때 상이한 것을 알 수 있다. 이는 valence와 arousal 등급이 동일한 상황에서 감정을 표현하는데 불연속 값을 가지기 때문이다. 비교를 위해, 본 발명은 모든 참가자들에 대해 0부터 6까지 스케일된 정서적 라벨 쌍의 평균 제곱 오차 제곱근(RMSE; root mean squared errors)을 계산한다. 여기서, 정서적 라벨 쌍의 평균 제곱 오차 제곱근은 아래 <수학식 12>와 같이 계산될 수 있다.5 shows different parameters

,

and

The results show the results of the 5 times cross-effectiveness scheme for valence and arousal rating distance between SAM and the framework of the present invention. It can be seen that the emotional curve is different when compared to the SAM-grade pair because it expresses affective dynamics in spatial and temporal situations and includes multiple pairs of emotional labels. This is because the valence and arousal grades have a discontinuity in expressing emotion in the same situation. For comparison, the present invention calculates the root mean squared errors (RMSE) of a pair of emotional labels scaled from 0 to 6 for all participants. Here, the square root of the mean square error of the emotional label pair can be calculated as shown in <Equation 12>.

[Equation 12]

는 예측된 정서적 라벨의 쌍을 의미하고,

는 상황

에서 실측 값 라벨의 쌍을 의미할 수 있다. 0은 부정적 밸런스 등급을 표현하고, 3은 중립 밸런스 등급을 표현하며, 6은 긍정적 밸런스 등급을 표현한다. 또한, 0은 중립 어라우절 등급을 표현하며, 3은 낮은 어라우절 등급을 표현하며, 6은 높은 어라우절 등급을 표현한다.here,

Means a pair of predicted emotional labels,

The situation

In can mean a pair of measured value labels. 0 represents a negative balance grade, 3 represents a neutral balance grade, and 6 represents a positive balance grade. In addition, 0 represents a neutral aura level, 3 represents a low aura level, and 6 represents a high aura level.

와

가 어라우절 값 A(k)를 결정하는 반면, 밸런스 값 V(k)는

와

파라미터에 의해 결정된다. 파라미터

은 적응 구성 요소 l(k)가 0이 되는 경우 결정된다. 이는 밸런스 값 V(k)의 부호와 어라우절 값 A(K)의 증가에 직접적으로 영향을 준다.

가 더 작은 값을 가지게 되면, 적응 구성 요소 l(k)는 0이 되고 밸런스 값 V(k)의 부호는 앞선 k에서 양이 된다. 도 7a에 도시된 바와 같이, 밸런스 등급에 대한 최고 성능은 0.5이고, 두 개의 서로 다른 포인트 0.25와 0.75은 밸런스 등급에 대하여 그 다음 최고 파라미터이다. 반대로, 도 7b에 도시된 바와 같이 어라우절 등급의 거리는 대부분의 참가자에 대해 0.5 이후에 최소화된다. 파라미터

과

는 모션 구성 요소와 적응 구성 요소에 관한 어라우절 레벨을 결정한다. 그 결과, 본 발명의 시스템은 도 7c에 도시된 바와 같이 파라미터

에 대하여 대부분의 참가자에 대해 0.7과 0.8 사이에서 어라우절 등급에 대하여 최고의 성능을 가지는 것을 알 수 있다.

이 0.85보다 크고 0.4보다 작은 경우 범위를 벗어난 거리는 거의 동일하게 유지된다. 파라미터

과

는 밸런스 레벨을 결정한다.

이 더 큰 값을 가지게 되면 그 결과 밸런스는 접근-회피 행동과 연관된 감정을 표현할 수 없다. 도 7d에 도시된 바와 같이, 본 발명에 따른 시스템의 성능은 파라미터

이 변하는 경우 현저하게 변동하는 것을 알 수 있다. 본 발명에서 모든 참가자에 대해

과

을 0.75와 0.5로 설정하고,

은 실험에서 각 개인에 대해 최소 거리를 만들기 위하여 0.25, 0.5, 0.75에서 선택될 수 있다.

Wow

Determines the aura value A(k), while the balance value V(k)

Wow

It is determined by parameters. parameter

Is determined when the adaptive component l(k) becomes zero. This directly affects the sign of the balance value V(k) and the increase in the aura clause value A(K).

If has a smaller value, the adaptive component l(k) becomes 0 and the sign of the balance value V(k) becomes positive in the preceding k. As shown in Fig. 7A, the highest performance for the balance class is 0.5, and two different points 0.25 and 0.75 are the next highest parameters for the balance class. Conversely, as shown in FIG. 7B, the distance of the aura level is minimized after 0.5 for most participants. parameter

and

Determines the aura level for the motion component and the adaptive component. As a result, the system of the present invention is a parameter as shown in Figure 7c

For most participants, it is found that it has the best performance for the aura level between 0.7 and 0.8.

If it is greater than 0.85 and less than 0.4, the out-of-range distance remains almost the same. parameter

and

Determines the balance level.

With this larger value, the resulting balance cannot express the emotions associated with approach-avoidance behavior. As shown in Figure 7d, the performance of the system according to the invention is a parameter

It can be seen that when it changes, it fluctuates remarkably. For all participants in the present invention

and

Set to 0.75 and 0.5,

Can be selected from 0.25, 0.5, and 0.75 to create the minimum distance for each individual in the experiment.

Preprocessing and setting of EEG data for classification

In the pre-treatment process, a high-pass filter with a 2 Hz cutoff frequency using the EEGlab toolbox and the same blind source separation technique for removing eye artifacts are applied. A constrained independent component analysis (cICA) algorithm is applied to refine the signal by removing motion artifacts. The cICA algorithm is an extended algorithm of ICA.

EEG signals are vulnerable to motion artifacts. The present invention can improve the quality of the EEG signal by abandoning or removing the EEG signal having a high correlation with the motion artifact, instead of separating and removing the motion artifact from the EEG signal generated by body movement. To this end, the present invention allows the EEG signals to be divided into two separate groups by changing the accelerometer data G() in a range from 0 to 1. Features are extracted from each of the two separate groups, for example, average power, maximum amplitude, standard deviation of amplitude, and amplitude as Kurtosis and amplitude skewness. These extracted features are matrixed to describe the main characteristics of the enlightened EEG. PCA (principal component analysis) is used to express the features in a two-dimensional space, and then the batacharyya distance between two groups is calculated in the two-dimensional space. Optical G() is a clean EEG and a contaminated EEG-like differentiator, determined by the maximum distance between the two groups. To analyze physiological and psychological relationships in the present invention, EEG spectral power, three components, and emotional labels are used in an emotional situation in which the accelerometer data G() is less than an optimal value.

In order to study the efficiency of the emotional label obtained from the emotional curve using the EEG signal, the present invention provides the emotional curve with seven states in the balance-aura space, for example, low-arousal/ LANV. negative valence), low-arousal/neutral balance (LAUV), low-arousal/positive valence (LAPV), mid-arousal/negative balance (MANV; mid-arousal) /negative valence), mid-arousal/positive valence (MAPV), high arousal/negative valence (HANV) and high arousal/negative valence (HAPV) /positive valence). Here, since the present invention is not characterized in actual experiments, MANV and HAUV can be excluded.

Some studies that extract EEG-based features from emotion recognition categorize them into three domains: time, time frequency, and frequency. Assuming that the signal is fixed during the psychological period, the frequency domain feature is the most used of the three domains. Thus, the present invention uses frequency domain features in different frequency bands, such as higher order spectra (HOS), power spectral density (PSD).

의 평균을 사용한다. 바이코히어런스는 신호 x(t)의 정규화된 바이스펙트럼

일 수 있다. 신호들은 겹치지 않는 세그먼트들로 분할되고, 각 세그먼트 내에서 데이터는 해닝(Hanning) 윈도우 및 푸리에 변환된다. 여기서, 바이스펙트럼은 아래 <수학식 13>과 같이 정의될 수 있다.The HOS feature is used to analyze human emotions by spectral representation of higher order moments or cumulants of the signal. In particular, the present invention, in order to study the efficacy of the emotional label obtained from the emotional curve using the EEG signal, four frequency bands, for example, theta (4-7 Hz), alpha (8-13 Hz), beta (14-29 Hz) , Bi-coherence in the frequency band of gamma (30-45 Hz)

Use the average of. The bicoherence is the normalized bispectrum of the signal x(t).

Can be The signals are divided into non-overlapping segments, and the data within each segment is transformed to a Hanning window and Fourier. Here, the bi-spectrum can be defined as <Equation 13> below.

[Equation 13]

는 신호 x(t)의 푸리에 변환을 의미하고,

는

의 복소 공액(complex conjugate)을 의미할 수 있다. 바이스펙트럼은 신호의 다른 구성 요소의 위상 정보를 보전한다. 두 주파수 구성 요소

과

는

의 주파수에서 세번째 구성 요소가 존재하는 경우 위상으로 결합된다. 바이코히어런스

는 아래 <수학식 14>와 같이 정의될 수 있다.here,

Is the Fourier transform of the signal x(t),

The

It may mean a complex conjugate of. Bispectrum preserves the phase information of other components of the signal. Two frequency components

and

The

At the frequency of the third component is present, the phases are combined. Bicoherence

Can be defined as <Equation 14> below.

[Equation 14]

는

에서 파워 스펙트럼을 의미할 수 있다. 바이코히어런스는 두 주파수 구성 요소 간의 위상 커플링 범위를 정량화한다. 결과 주파수 분해능은

과

축 모두에서 1Hz이다. 네 개의 주파수 대역에서 바이코히어런스

의 평균 크기는 아래 <수학식 15>와 같이 계산될 수 있다.here,

The

Can mean power spectrum. Bicoherence quantifies the range of phase coupling between two frequency components. The resulting frequency resolution

and

1 Hz on both axes. Bicoherence in four frequency bands

The average size of can be calculated as <Equation 15> below.

[Equation 15]

는 두 주파수 대역 q1과 q2에서 주파수 구성 요소들을 의미할 수 있다. PSD의 파워 특징은 Welchs 방법을 이용하여 추정되고 네 개의 주파수 대역으로 나누어진다.

와 네 개 주파수 대역의 평균 파워는 EEG 신호를 이용한 정서 커브로부터 획득된 정석적 라벨의 상관을 분석하는데 사용된다.Here, q1 and q2 mean frequency bands,

Can mean frequency components in two frequency bands q1 and q2. The power characteristics of the PSD are estimated using the Welchs method and divided into four frequency bands.

And the average power of the four frequency bands are used to analyze the correlation of the qualitative label obtained from the emotional curve using the EEG signal.

FIG. 6 shows an example of an emotional curve combining an arousal curve and a balance curve, and each curve illustrated in FIG. 6 expresses an emotional expression of an emotional situation in a participant's daily life, and the average curve of a parabolic shape is neutral It covers the VA emotion space, with the exception of some emotions characterized by balance and high-level aura clauses. Here, the dotted line represents the emotional curve for the individual participants, and the solid line represents the average curve for all participants.

Physiological specificity : The ANOVA test results for the bicoherence magnitude and PSD in four frequency bands in seven emotional states have a p value of 0.0679 in the beta frequency band and a p value lower than 0.05 in the other frequency bands. The p-value is due to the magnitude of bicoherence in all frequency bands, and in theta, alpha, and gamma frequencies, the three frequency bands appear significantly different from the seven emotional states. These results indicate that PSD and bicoherence can be useful as physiological features that classify emotions.

Based on the results of ANOVA, the present invention can design a classification process that can evaluate the efficiency of the emotional curve well so that it can act as a reliable emotional label in EEG-based emotion recognition. In particular, this name is SVM (support vector machine) with a fifth-order polynomial kernel that extracts PSD and bicohererson features from four frequency bands, uses mutual information for feature selection, and generates the highest accuracy in recognition. ) The selected features are classified into seven emotional states using an algorithm. The effectiveness of the emotional curve as a reliable emotional label for distinguishing different emotional states can be evaluated by thematic classification performance.

FIG. 7 shows an exemplary diagram of the results of the classification procedure through the 5-fold cross-effectiveness scheme for all participants, and shows the average accuracy of all individual results for the seven emotional states.

As shown in Fig. 7, the accuracy for recognizing the seven emotional states is 66.65%, which means that the EEG signals grouped by the emotional labels have distinct patterns. The LAUV state has the lowest standard deviation of 3.7 and the highest accuracy, and the LAPV state and LANV state have relatively higher standard deviations of 7.09 and 7.17 than the other five states. This means that participants have different activation patterns in the LAPV and LANV states, while similar patterns of brain activity, such as feeling calm and relaxed in the LAUV state.

Physiological characteristics : The present invention investigates the statistical relationship between EEG spectral power and emotional labels in four frequency bands to reduce the gap between psychological measurements and physiological evidence. Through Table 1 below, it can be seen that the alpha frequency component has a higher correlation for the two classes.

Here, F3 and F4 may mean two electrodes for detecting the EEG signal.

The emotional data set of the present invention includes the hand movements of the participants, and since the Mu rhythm (8-11 Hz) activated by movement in the motor cortex region has a strong relationship with the alpha frequency component, the movement cortex activation related to movement is It is correlated with the balance level or aura level.

However, this does not mean that the psychological measurement of hand movements is not related to biological changes. To determine if the alpha frequency band is contaminated by hand movements or related to psychological measurements, the present invention calculates a correlation coefficient between the powers of the four parking bands and the three components. Table 2 below shows the correlation for each component between physiological brain activity and emotional state.

As described above, the motion component and the synchronous component are affected by movement, but only the synchronous component reflects the user's approach behavior in psychology and quantifies it in the balance class. The association between the physiological change in the alpha frequency band and the psychological measurement of motion along with the coefficient values in the alpha frequency band associated with the motion component and the synchronous component cannot be the result of motor cortex activation. This is because Mu Rhythm is related to activation and positively related only to the synchronous component, not the motion component.

Moreover, the present invention makes it clear whether the emotional label includes physiological features due to psychological phenomena. To investigate the relationship between EEG spectral power and the three components, the present invention uses the Granger causality test. Here, the Granger causality test can analyze the flow of information between the two series. It can be said that if the pattern at x1 is repeated at x2 within a certain period of time, the x2 granger causes x1. At this time, the present invention can use a multivariate Granger causality test that is fit by a vector autoregressive time series.

The present invention calculates Granger causality with EEG spectral power in four frequency bands from three components for all participants. It can be seen that the mean 0.29, 0.36 and 0.08 of the causality test have positive values for the EEG spectral power under the influence of the three components, respectively. The alpha frequency band and theta frequency band have a relatively high causality and are influenced by all components compared to other components. Conversely, the beta frequency band has a low causality under both components.

In other emotional labels, the characteristics of brain signals are analyzed by correlation and causality tests, and EEG-based statistical analysis shows that physiological responses correlate with successive emotional labels.

As such, the framework according to the present invention performs emotional context expression and modeling, and can provide the following contributions.

(1) Emotional situation expression: The present invention proposes an emotional curve. The emotional curve is a set of accumulated points for the balance and the emotion space over time, and represents the emotional dynamics in the real environment.

(2) Emotional situation modeling: In the present invention, when an image sequence is given *?*, an expected feeling is sensed and a change in the situation is tracked. The present invention proposes three components to model changes in the situation: motivation, motion and adaptation, which can be calculated with low-level visual features extracted to reflect the emotional response to the situation. In particular, motivational components are based on tendency to avoid access and quantify *?* feelings of behavioral tendencies from emotional behaviors that arise in response to emotional content in context.

(3) Physiological experiments to verify the effects of emotional labels: The present invention is evaluated using a series of long-term life records for several days in real scenarios, and investigates the characteristics of brain signals for different emotional labels, Analyze. Statistical analysis based on electroencephalography (EEG) shows that physiological responses are consistently related to emotional labeling.

The framework according to the present invention may be implemented as an emotion estimation system. For example, the emotion estimation system according to an embodiment of the present invention may include a calculation unit, a modeling unit, a generation unit, and an estimation unit. Of course, the configuration means here may vary depending on the situation.

The calculator calculates a motion component, a motivation component, and an adaptation component through learning the user's affective situation using an image sequence.

At this time, the calculation unit may calculate the motion component by characterizing and quantifying motion of an emotional object between adjacent frames using an optical flow estimation for an image sequence.

Furthermore, the calculator may calculate motion components by removing motion noise using motion activity estimated in each image frame of the image sequence and accelerometer data collected by the accelerometer attached to the user's body.

Furthermore, the calculator may acquire a protruding area corresponding to a user's interest in an image frame of an image sequence, and calculate the synchronous component based on the obtained protruding area.

Furthermore, the calculation unit may calculate the synchronous component by calculating divergence and rotation using an optical flow of a predetermined region around the target of interest in the image frame of the image sequence.

Furthermore, the calculation unit may calculate the adaptive component based on the emotional situation length accumulated up to the current frame of the image sequence that changes over time.

The modeling unit models a balance model and an arousal model based on the calculated motion component, synchronous component, and adaptive component.

At this time, the modeling unit may model the aurajeol model using the calculated motion component and the adaptive component, and may model the balance model using the aurajeol model and the synchronous component.

The generation unit generates an emotional curve based on the emotional situation expressed on the emotional space by the modeled balance model and the aurael model.

The estimation unit estimates the emotion value of the user using the emotion curve.

In addition, the system according to another embodiment of the present invention models a balance model and an arousal model based on a user's behavior using an image sequence, and is evaluated by the modeled balance model and the aurael model By generating an emotional curve based on an emotional situation expressed in space, the emotional value of the user may be estimated using the generated emotional curve.

Here, the system can detect the user's interest from the image sequence, and model a balance model and an arousal model based on the interaction with the detected interest.

Although the description of the system of the present invention is omitted, it is apparent to those skilled in the art that the system according to the present invention can include all the contents described in FIGS. 3 to 7 above.

The system or device described above may be implemented with hardware components, software components, and/or combinations of hardware components and software components. For example, the systems, devices, and components described in embodiments include, for example, processors, controllers, arithmetic logic units (ALUs), digital signal processors (micro signal processors), microcomputers, and field programmable arrays (FPAs). ), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose computers or special purpose computers. The processing device may run an operating system (OS) and one or more software applications running on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For convenience of understanding, a processing device may be described as one being used, but a person having ordinary skill in the art, the processing device may include a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that may include. For example, the processing device may include a plurality of processors or a processor and a controller. In addition, other processing configurations, such as parallel processors, are possible.

The software may include a computer program, code, instruction, or a combination of one or more of these, and configure the processing device to operate as desired, or process independently or collectively You can command the device. Software and/or data may be interpreted by a processing device or to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. , Or may be permanently or temporarily embodied in the transmitted signal wave. The software may be distributed on networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiments may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer-readable medium may include program instructions, data files, data structures, or the like alone or in combination. The program instructions recorded on the medium may be specially designed and constructed for the embodiments or may be known and usable by those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs, DVDs, and magnetic media such as floptical disks. -Hardware devices specifically configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language code that can be executed by a computer using an interpreter, etc., as well as machine language codes produced by a compiler. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described by a limited embodiment and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques are performed in a different order than the described method, and/or the components of the described system, structure, device, circuit, etc. are combined or combined in a different form from the described method, or other components Alternatively, even if replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (22)

Calculating a motion component, a motivation component, and an adaptation component through learning of the user's affective situation using an image sequence in the calculator;
Modeling a balance model and an arousal model based on the calculated motion component, synchronous component and adaptive component in the modeling unit; And
Generating an emotional curve based on the emotional situation expressed on the emotional space by the modeled balance model and the aurael model in the generation unit
Emotion estimation method comprising a.
According to claim 1,
The calculating step
Emotion estimation method characterized by calculating the motion component by characterizing and quantifying the motion of an emotional object between adjacent frames using optical flow estimation for the image sequence.
According to claim 1,
The calculating step
Emotion estimation method comprising calculating motion components by removing motion noise using motion activity estimated in each image frame of the image sequence and accelerometer data collected by accelerometer attached to the user's body. .
According to claim 1,
The calculating step
A method for estimating emotions, wherein a protruding area corresponding to a user's interest is obtained from an image frame of the image sequence, and the synchronous component is calculated based on the obtained protruding area.
According to claim 4,
The calculating step
Emotion estimation method, characterized in that for calculating the synchronous component, by calculating the divergence and rotation using the optical flow of a certain area around the object of interest in the image frame of the image sequence.
According to claim 1,
The calculating step
A method for estimating emotions, wherein the adaptive component is calculated based on an emotional situation length that is a cumulative sum of emotional situations accumulated up to the current frame of the image sequence that changes over time.
According to claim 1,
The modeling step
Emotion estimation method, characterized in that for modeling the aurajeol model using the calculated motion component and the adaptive component.
According to claim 1,
The modeling step
Emotion estimation method, characterized in that for modeling the balance model using the aurajeol model and the synchronous component.
Modeling a balance model and an arousal model based on the user's behavior using the image sequence in the modeling unit;
Generating an emotion curve based on the emotional situation expressed on the emotional space by the modeled balance model and the aurael model in the generation unit; And
Estimating the emotion value of the user using the emotional curve in the estimation unit
Including,
The modeling step
Calculating a motion component, a motivation component, and an adaptation component based on the user's behavior; And
Modeling the balance model and the arousal model based on the calculated motion component, synchronous component and adaptive component
Emotion estimation method comprising a.
delete delete A calculation unit that calculates a motion component, a motivation component, and an adaptation component through learning the user's emotional situation using an image sequence;
A modeling unit that models a balance model and an arousal model based on the calculated motion component, synchronous component, and adaptive component; And
Generating unit for generating emotional curves based on the emotional situation expressed on the emotional space by the modeled balance model and auraun model
Emotion estimation system comprising a.
The method of claim 12,
The calculation unit
Emotion estimation system characterized by calculating the motion component by characterizing and quantifying motion of an emotional object between adjacent frames using optical flow estimation for the image sequence.
The method of claim 12,
The calculation unit
Emotion estimation system characterized by calculating motion components by removing motion noise using motion activity estimated in each image frame of the image sequence and accelerometer data collected by accelerometer attached to the user's body .
The method of claim 12,
The calculation unit
Emotion estimation system, characterized in that for acquiring a protruding area corresponding to a user's interest in an image frame of the image sequence, and calculating the synchronous component based on the obtained protruding area.
The method of claim 15,
The calculation unit
Emotion estimation system, characterized in that for calculating the synchronous component by calculating the divergence and rotation using the optical flow of a predetermined area around the object of interest in the image frame of the image sequence.
The method of claim 12,
The calculation unit
Emotion estimation system characterized by calculating the adaptive component based on the emotional situation length, which is the cumulative sum of emotional situations accumulated up to the current frame of the image sequence that changes over time.
The method of claim 12,
The modeling unit
Emotion estimation system, characterized in that for modeling the aurajeol model using the calculated motion component and the adaptive component.
The method of claim 12,
The modeling unit
Emotion estimation system, characterized in that for modeling the balance model using the aurajeol model and the synchronous component.
A modeling unit that models a balance model and an arousal model based on a user's behavior using an image sequence;
A generation unit generating an emotion curve based on the emotional situation expressed on the emotional space by the modeled balance model and the auraun model; And
Estimator for estimating the emotion value of the user using the emotional curve
Including,
The modeling unit
Calculate a motion component, a motivation component and an adaptation component based on the user's behavior, and based on the calculated motion component, synchronous component and adaptation component Emotion estimation system that models the balance model and the arousal model.
delete delete

Images