KR20190125707A - Method for estimating emotion based on psychological activity and biosignal of user and system therefor - Google Patents

Abstract

Disclosed are a method for estimating emotion based on psychological activity and biosignal of a user and a system thereof. According to an embodiment of the present invention, the method for estimating emotion includes the following steps: calculating a motion component, a motivation component, and an adaptation component through learning an affective situation of a user by using an image sequence; modeling a balance model and an arousal model based on the calculated motion component, motivation component, and adaptation component; and generating an emotion curve based on the affective situation represented on an emotion space by the modeled balance model and arousal model.

Description

Methods for estimating emotion based on psychological activity and biosignal of user and system therefor}

The present invention relates to a technique for inferring a user's emotion, and more particularly, to a method and a system for estimating a user's emotion based on a user's psychological activity and a biosignal.

Emotional awareness has received a lot of attention lately because it makes it easier to understand the emotional interaction between a person and a computer by measuring emotional states such as joy, excitement, and fear. In order to model a person's emotional state, subjective self-reports that describe in detail the feeling a person may provide useful information that can be used to indicate (or label) his emotional state. For example, the emotional assessment tool, Self-Assessment Manikin (SAM), uses a graphical scale that depicts a cartoon character that expresses three emotional elements: joy, arousal, and dominance. In most cases, the participant's self-assessment is performed in a laboratory setting in which the participant is seated in a comfortable chair to indicate the level of evaluation. This method of evaluation has a problem with validity and confirmation that cannot be performed in outdoor environments other than answering the participant's exact answer to what he or she feels and instead expecting others to answer. In addition, this assessment method can yield immediate human emotions for complex emotional conditions and only provide a limited understanding of emotional dynamics in everyday life. Therefore, it is important to provide an automatic method for displaying human emotions triggered in real situations.

However, little research has been done on affective labeling of real situations. It is difficult to quantify emotional responses based on an understanding of a situation. A cognitive understanding of the actual objects with which humans interact is needed, which can determine the emotional level of emotions expected based on interactions. The most difficult problem is to quantify the level of object-dependent emotions, find visual attention in the human cortex, and reduce artifacts in the real environment.

Providing emotional labels can enhance specializations, such as content recommendation systems, and quantify the subject's emotional response to the stimulus that is the cause of the emotional dimension by using both explicit and implicit tagging.

Human emotion can be conceptualized in three important senses: Valence (V), Arousal (A), and Dominance (D). Valence is a type of emotion and characterizes an emotional state or reaction, from discomfort or negative emotions to pleasant, happy, positive emotions. Arousal is the intensity of emotion, which characterizes an emotional state or reaction from drowsiness or boredom to extreme excitement. Dominance distinguishes emotional states with valence and arousal from "no control" to "full control". For example, sadness and anger have similar valance and arousal values, but are distinguished from dominance values. The full range of human emotion can be expressed as a series of points in the three-dimensional VAC coordinate space. Conversely, each basic emotion can be expressed as a bipolar entity, characterized by all emotions by valence and arousal, as shown in FIG. 1A, and emotion labels can be expressed at various locations on the two-dimensional VA plane.

In addition, according to various conventional studies, it can be seen that the emotional response mapped to the emotional coordinate system is approximately parabolic as shown in FIG. 1B. In one example prior art, the nature of an agent is defined by using parabolic surfaces to assign temperament, mood and emotion to the emotional agent.

The explicit approach provides labels by requiring users to report their feelings in response to a given event or stimulus. The International Affective Picture System (IAPS) is a common dataset used to obtain sentiment labels using explicit self-reporting tools such as SAM. As shown in Figure 2, SAM measures a wide range of emotional responses to various stimuli associated with valence (positive to negative), arousal (high to low level) and dominance (low to high level). Is a picture-based evaluation technique. Emotional labels obtained from explicit self-reporting tools are considered actual data on the emotional state and are used to build a reliable emotion recognition system.

However, the main disadvantage of the explicit approach to expressing human emotion is the intrusiveness of the reporting process. In contrast, the implicit emotional labeling approach is a non-intrusive measure. Labeling can be obtained by exposing the user to a stimulus and recording the response to the exposure. Visual and motion-related functions are important elements in emotion recognition. The technique of one conventional embodiment used facial change characteristics to label human emotions, and the technique of another conventional embodiment studied a target motion as a visual feature in response to human emotion, and emotional arousal as the intensity of motion increased. It has been shown that levels can increase. The technique of another conventional embodiment has developed a method of characterizing arousal using motion intensity and shot change rate, while the technique of another conventional embodiment uses motion activity to determine the arousal level and the continuous The change is represented by a curve.

Another method is to extract emotional features of psychological behaviors and associate them with emotional states. Many conventional studies have suggested methods for measuring emotional behaviors. For example, arm movements such as flexion and extension have been used to investigate positive and negative interactions between responses to approach and avoidance behaviors. One prior art technique used joysticks to determine if positive and negative stimuli promoted approach and withdrawal behavior. Participants were instructed to control the joystick by pulling the joystick to increase the size of the stimulus or by pushing the joystick to reduce the size of the stimulus, and their metrics could distinguish adolescents' approach and avoidance behaviors in response to positive or negative stimuli. However, the technology is limited to controlled experimental setups and uses some limited perceptual work that requires the use of certain equipment and does not allow participants to interact with the system in real time.

Embodiments of the present invention provide a method and system for estimating a user's emotion based on the user's psychological activity and bio-signal.

Embodiments of the present invention provide a method and system for extracting a value of an emotion felt by a user based on the user's behavior.

In the emotion estimation method according to an embodiment of the present invention, a motion component, a motivation component, and an adaptation component are obtained through learning of an emotional situation of a user using an image sequence. Calculating; Modeling a balance model and an arousal model based on the calculated motion component, synchronization component, and adaptive component; And generating an emotional curve based on the emotional situation represented on the emotional space by the modeled balance model and the augmentation model.

The calculating may calculate the motion component by characterizing and quantifying the motion of the emotional object between adjacent frames using optical flow estimation for the image sequence.

The calculating may include calculating motion components by removing motion noise using motion activity estimated in each image frame of the image sequence and accelerometer data collected by an accelerometer attached to the user's body.

The calculating may include obtaining a protrusion area corresponding to a user's interest in the image frame of the image sequence, and calculating the sync component based on the acquired protrusion area.

The calculating may include calculating the synchronization component by calculating divergence and rotation using an optical flow of a predetermined region around the object of interest in the image frame of the image sequence.

The calculating may calculate the adaptive component based on an emotional situation length accumulated up to a current frame of the image sequence that changes over time.

The modeling may be performed by using the calculated motion component and the adaptive component to model the AU clause model.

In the modeling, the balance model may be modeled using the augmentation model and the synchronization component.

In accordance with another aspect of the present invention, an emotion estimation method includes: modeling a balance model and an arousal model based on a user's behavior using an image sequence; Generating an emotional curve based on the emotional situation represented on the emotional space by the modeled balance model and the ʻauraud 'model; And estimating an emotion value of the user using the emotional curve.

The modeling may include calculating a motion component, a motivation component, and an adaptation component based on the user's behavior; And modeling a balance model and an arousal model based on the calculated motion component, synchronization component, and adaptive component.

The modeling may include detecting an object of interest of the user from the image sequence, and modeling the balance model and the arousal model based on the interaction with the detected object of interest.

The emotion estimation system according to an embodiment of the present invention is a motion component, a motivation component, and an adaptation component through learning of an emotional situation of a user using an image sequence. A calculation unit for calculating a); A modeling unit for modeling a balance model and an arousal model based on the calculated motion component, synchronization component, and adaptive component; And a generator configured to generate an emotional curve based on the emotional situation represented on the emotional space by the modeled balance model and the augmentation model.

According to another aspect of the present invention, an emotion estimation system includes a modeling unit for modeling a balance model and an arousal model based on a user's behavior using an image sequence; A generator configured to generate an emotional curve based on the emotional situation represented in the emotion space by the modeled balance model and the ʻauraud 'model; And an estimator estimating the emotion value of the user using the emotional curve.

According to embodiments of the present invention, it is possible to provide a calculation framework for providing affective labels in actual situations.

Existing emotion recognition systems allow users to passively value their emotions based on a self-assessment table called self-assessment and recognize the emotions accordingly, while the present invention provides for emotions from the user's behavior without such a self-assessment table. The value can be calculated automatically.

In addition, the present invention can induce the automation of the existing emotion recognition system because it infers the emotion value through the behavior of the user in the real life, and can be applied to various applications in the real life environment.

The present invention can be applied to the medical field through the application of the early diagnosis technology for depression, a technology for identifying the emotional response of the user to the content in the virtual / augmented reality.

In addition, the present invention can be extended to the wearable market according to the miniaturization technology of the sensors for measuring brain waves, and can be used in daily life through the production of brain wave measuring wearable devices that can be linked to the user's smartphone.

Figure 1 shows an example of the distribution of emotion and parabolic shape in the balance-arranged space.
2 illustrates an exemplary diagram for explaining a self-assessment manikin (SAM).
3 shows a conceptual diagram of a framework according to an embodiment of the present invention.
4 illustrates an exemplary diagram for describing a synchronization component.
FIG. 5 shows the results of a 5-fold cross-validation scheme on the balance between the SAM and the framework of the present invention and the augmentation rating distance for different parameters.
FIG. 6 shows an example of an emotional curve combining an augmentation curve and a balance curve.
FIG. 7 shows an exemplary view of the results of the classification procedure with a 5-fold cross validity scheme for all participants.

Advantages and features of the present invention and methods for achieving them will be apparent with reference to the embodiments described below in detail with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in various different forms, only these embodiments are intended to make the disclosure of the present invention complete, and common knowledge in the art to which the present invention pertains. It is provided to fully inform the person having the scope of the invention, which is defined only by the scope of the claims.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, “comprises” and / or “comprising” refers to a component, step, operation and / or element that is one or more of the other components, steps, operations and / or elements. It does not exclude existence or addition.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used in a sense that can be commonly understood by those skilled in the art. In addition, terms that are defined beforehand that are generally used are not to be interpreted ideally or excessively unless they are clearly specifically defined.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. The same reference numerals are used for the same components in the drawings, and duplicate descriptions of the same components are omitted.

Embodiments of the present invention provide a low-level, feature-based framework for modeling emotional video content, such as two-dimensional emotional coordinate spaces, for example, a valence-arousal emotional coordinate space. The idea is to provide a framework for expressing and modeling emotional situations in a series of points.

Here, the present invention can detect visual interest in the cortex of a human being, which is the basis of psychological behavior, and quantify anticipated emotions based on interactions with objects of interest.

The present invention can express the result of a spatiotemporal situation as a polynomial curve called an emotional curve.

Here, the video can be expressed as an emotional curve that stably describes the transition expected from one feeling to the other while watching the video, and the emotional curve can be represented by the International Affective Picture System (IAPS) (PJ Lang, MM Bradley, and BN Cuthbert, "International affective picture system (iaps): Technical manual and affective ratings," NIMH Center for the Study of Emotion and Attention, pp. 39-58, 1997.) and International Affective Digitized Sounds System (IADS) Bradley and P. Lang, "International affective digitized sounds (iads): Technical manual and affective ratings," Center for Research in Psychophysiology, Univ. Florida, Gainesville, FL, 1991.) Can be based.

IAPS and IADS include the measurement of emotional responses to calibrated audiovisual stimuli, although both systems aim to collect a wide range of emotions in audiovisual content, but the emotional response mapped to the emotional coordinate system is roughly parabolic. This has been determined through several studies. For example, parabolic surfaces can be used to define an agent's personality by assigning temperament, mood, and emotion to an emotional agent.

The present invention derives a model that derives an emotional curve from low-level features extracted from an image sequence similar to the above, wherein the model in the present invention reflects the emotional behavior of humans based on the events of life and the emotional situation. Approach-withdrawal theory is used to calculate the emotional curve representing

In the framework of the present invention, an emotional situation is referred to as an 'emotional curve'. Here, the emotional curve may mean a polynomial curve that is fitted to a set of points in the valence-arousal emotional space. In addition, in order to model and express an emotional situation in a real environment, environmental information may be collected using a device that can be easily worn so that a user can behave freely in everyday situations. The device may consist of a front camera, an accelerometer and a small physiological sensor. The present invention uses the data collected in such devices to learn emotional contextual representations and to provide appropriate emotional labels for physiological changes that can be used to classify human emotions.

The emotional context in the present invention may be defined as a specific arrangement of emotional entities, and the present invention provides a framework for modeling and expressing such emotional situations. Here, the emotional reality may be an object of various real worlds where people meet, interact, and feel emotional responses.

When the emotional situation is modeled by the present invention, the contents of human interaction can be understood, and expressing such a situation can determine the expected level of emotion based on the interaction. Thus, the framework of the present invention can help to reduce semantic differences between cognitive and emotional emotions in real situations.

The present invention will be described in detail as follows.

3 illustrates a framework in accordance with one embodiment of the present invention, wherein the framework of the present invention provides for emotional labeling of an image sequence in a practical scenario.

The framework of the present invention learns and expresses an emotional situation based on an egocentric image to quantify the feeling of behavioral trends from the emotional behavior faced by the emotional content in the context. The present invention can take a self-centered image every time frame, use auxiliary accelerometer data as input, and output two emotion points in valence-arousal space.

Here, the learned points may be represented by polynomial curves called emotional curves.

As shown in FIG. 3, the framework of the present invention expresses an emotional situation using three components calculated after calculating a motion component, a synchronization component, and an adaptive component from an image sequence, and a two-dimensional emotion space. Emotional curves are generated using the emotional situations represented above, and emotional labeling is provided using the generated emotional curves.

Emotional situation learning

One prior art technique (A. Hanjalic and L.-Q. Xu, "Affective video content representation and modeling," IEEE transactions on multimedia, vol. 7, no. 1, pp. 143-154, 2005.) We developed an emotional curve for valence and arousal spaces based on three criteria used to psychologically justify The present invention modifies the aforementioned criteria in order to apply the framework of the present invention in practical situations.

(1) Comparability: Comparable values of valence, arousal and resultant emotional curves obtained in different situations for similar types of emotional behavior. This criterion naturally imposes standardization and scaling requirements when calculating time curves.

(2) Compatibility: The shape of the emotional curve reflects the situation given a certain time in the valence-arousal emotional space. When the situation is over, the shape of the curve becomes an approximate parabolic shape in the 2D emotion space.

(3) Smoothness: indicates the degree of emotion retention of the previous frame. This ensures that the emotional proportion of the content involved in eliciting human emotions does not suddenly change between successive frames of the situation.

1) Emotional Characteristics of Emotional Situation Learning

The present invention can observe various emotional expressions used in the context. The present invention uses the general functions A (k) and V (k) for arousal and valence in frame k to model these emotional phenomena as arousal and valence. Both functions may have a suitable form of function incorporating three components: the motion component of the emotional response, the motivation component of the emotional approach-avoidance behavior, and the adaptive component of the emotional situation.

a) motion component of the emotional response

Several previous studies have reported that motion is one of the most important visual features that affect an individual's emotional response. Motion is expressive and can induce strong emotional reactions. Studies on the effects of subject motion on human emotional responses show that arousal increases with increasing motion intensity.

The present invention calculates motion components to estimate motion activity m (k) in each video frame k.

의 평균 크기는 모션 활동성을 공식화하는데 사용될 수 있다.Here, the present invention can use optical flow estimation to characterize and quantify the motion of an emotional object between adjacent frames, all estimated motion vectors

The average size of can be used to formulate motion activity.

The motion activity m (k) may be expressed as Equation 1 below.

[Equation 1]

는 프레임 k에서의 모션 벡터 i를 의미하고, B는 프레임 k에서의 모션 벡터의 수를 의미할 수 있다.here,

May mean the motion vector i in the frame k, and B may mean the number of motion vectors in the frame k.

에 가속도계 데이터

를 사용할 수 있으며, 모션 컴포넌트

는 아래 <수학식 2>와 같이 나타낼 수 있다.Since self-centered images are typically captured by cameras attached to the human body, the images may contain motion noise that contaminates the motion activity m (k). The present invention provides a motion component to remove such motion noise.

Accelerometer data on

Can be used for motion components

Can be expressed as in Equation 2 below.

[Equation 2]

는 0과 1 사이에서 정규화된 가우시안 평활 결과를 의미할 수 있다.here,

May mean a Gaussian smoothing result normalized between 0 and 1.

는 모션 잡음이 증가하면 arousal이 감소한다는 것을 의미한다.Because these are not real factors of motion activity

Means that the arousal decreases as the motion noise increases.

4 illustrates a conceptual diagram for a motion component, as shown in FIG. 4, visual attention (or interest) from an image in each video frame k is detected in the human cortex by a protrusion map. By generating a binary protrusion map using a parabolic protrusion map detected from an image and generating a synchronization map using the generated binary protrusion map, an emotional approach-avoidance behavior associated with an object of interest can be calculated using the synchronization map. . That is, in the area covered by the binary derivation map, the emotional approach-avoidance behavior associated with the object of interest may be calculated using the optical flow around the object.

b) the motivational component of the emotional approach-avoidance behavior;

를 계산한다. 동기 구성 요소의 계산은 먼저 시각적인 주의(visual attention)을 통해 참가자의 의도를 파악하여 수행된다.Motivational components for modeling emotional behaviors that occur in response to emotional content in various situations, similar to motion components

Calculate The calculation of the motive component is first performed by grasping the participant's intention through visual attention.

Here, estimating the location of visual attention maintained at a particular fixed point may be performed with saliency prediction or detection.

Computer phenomena, robotics and neurosciences have seen this phenomenon and are getting more and more attention. The present invention may use a saliency model, for example, a saliency attentive model to obtain the most protruding region in the image frame.

Here, the salient model builds a convolutional LSTM with a set of features computed by multi-learned gaze priors with dilated convolutional networks (DCN) and salient target detectors. Human eye fixations can be predicted.

The present invention uses the final overhang map, which is a probability map with values within [0, 1], to produce a binary overhang map with a threshold value t _h . In a binary protrusion map, white is the main fixed area where participants visually care (or pay attention). The present invention learns the emotional approach-avoidance behavior associated with an object of interest by calculating the divergence and rotation using the optical flow around the object of interest in each video frame k, based on the projected object as a result of the human visual attention prediction. This approach is possible because accessing a single object is the same as enlarging the object and has the same effect on the divergence of the flow vector surrounding the center point of the protruding object.

In this case, the present invention may calculate the flow using multi-scale block based matching between adjacent frames.

는 아래 <수학식 3>과 같이 나타낼 수 있다.In order to describe the synchronous behavior, the present invention uses six raw optical flow patterns, e.g. rotation around the vertical axis, rotation around the horizontal axis, approach in the target direction, rotation around the optical axis of the image plane and complex hyperbolic flow (complex). Standardize flows with hyperbolic flows. Where, given the motion vector, the velocity at pixel c (x, y)

Can be expressed as in Equation 3 below.

[Equation 3]

는 픽셀 c₀에서의 속도를 의미하고,

는 아래 <수학식 4>와 같이 정의되는 행렬일 수 있다.here,

Is the velocity at pixel c ₀ ,

May be a matrix defined as in Equation 4 below.

[Equation 4]

는 아래 <수학식 5>와 같이 분해될 수 있다.At this time,

Can be decomposed as shown in Equation 5 below.

[Equation 5]

와

은 각각 발산 옵티컬 플로우와 회전 옵티컬 플로우를 의미하고,

과

는 상이한 유형의 쌍곡선 옵티컬 플로우를 의미할 수 있다.here,

Wow

Denotes a divergent optical flow and a rotating optical flow, respectively.

and

May mean different types of hyperbolic optical flow.

The matrices D, R, H ₁ and H ₂ may be as shown in Equation 6 below.

[Equation 6]

로 특징지어 질 수 있다. 본 발명은 관심 대상의 옵티컬 플로우 필드가 주어지면 상기 수학식 5와 최소 자승 오차 방법(least squared error method)을 이용하여 파라미터 벡터를 추정한다. 파라미터 u는 오른쪽 및 왼쪽 회전과 연관되고,

는 위쪽 및 아래쪽 방향과 연관되며, d는 대상 접근 및 회피와 연관되고, 나머지 세 파라미터 r, h₁, h₂는 결합된 모션을 나타낸다.The velocity of the motion vector field is approximately six parameters,

Can be characterized as Given an optical flow field of interest, the present invention estimates the parameter vector using Equation 5 and the least squared error method. Parameter u is associated with right and left rotation,

Is associated with up and down directions, d is associated with object approaching and avoidance, and the remaining three parameters r, h ₁ , h ₂ represent combined motion.

를 계산할 있으며, 아래 <수학식 7>과 같이 나타낼 수 있다.Sync component in frame k with six parameters

It can be calculated as shown in Equation 7 below.

[Equation 7]

Here, X and Y may mean the width and height of the optical flow field of interest.

c) the adaptation component of an emotional situation

를 도출함으로써, 상황의 감정적 적응을 모델링할 수 있으며, 함수

는 아래 <수학식 8>과 같이 나타낼 수 있다.The present invention uses situation lengths that change over time to represent a connection between the user's feelings and the intention to adapt to the situation. The present invention functions in frame k

By deriving, we can model the emotional adaptation of the situation and

Can be expressed as Equation 8 below.

[Equation 8]

및

은 함수

의 형상을 결정하고,

는 정서적 상황의 길이로, 현재 프레임 k까지의 정서적 상황의 누적 합계를 의미할 수 있다.here,

And

Is a function

Determine the shape of,

Is the length of the emotional situation, and may mean a cumulative sum of the emotional situations up to the current frame k.

는 현재 프레임 k의 상황이 이전 프레임 k-1과 동일하면 1만큼 값이 증가한다. 본 발명은 로그 함수 예를 들어, log₁₀ 함수를 사용하여 정서적 상황의 적응 구성 요소를 모델링할 수 있다.function

If the situation of the current frame k is equal to the previous frame k-1, the value is increased by one. The present invention can model the adaptive component of an emotional situation using a log function, for example, a log ₁₀ function.

2) Arousal and Valence Model

및 적응 구성요소

의 기여도를 통합하기 위하여 모션 및 적응의 가중된 평균을 사용한다. 상기 함수는 이동 평균 필터를 통해 구성 요소의 이웃하는 로컬 최대 값을 병합하기에 충분히 긴 평활화 윈도우(smoothing window)로 컨벌루션된다. 결과는 0과 1의 범위로 정규화된다. 여기서, 함수 A(k)는 아래 <수학식 9>와 같이 나타낼 수 있다.In order to model the arousal, the function A (k) is a motion component along an image sequence in an emotional situation at frame k.

And adaptive components

The weighted average of motion and adaptation is used to incorporate the contributions of. The function is convolved with a smoothing window long enough to merge the component's neighboring local maximum values through a moving average filter. The result is normalized to the range 0 and 1. Here, the function A (k) can be expressed as Equation 9 below.

[Equation 9]

는

인 두 함수에 가중치를 부여하기 위한 계수들을 의미할 수 있다.here,

Is

It may mean coefficients for weighting the two functions.

The compatibility criterion requires that the emotional curve generated by combining the arousal and valence time curves must include a region of the valence-arousal coordinate system that has a parabolic shape similar to the 2D emotion space. Here, the compatibility criterion may require an arousal value and an absolute value of valence associated with it.

Thus, the range of arousal values can determine the absolute value range of valence.

That is, the present invention can model valence by defining a function r (k) that captures the range dependency of this value in consideration of the value of arousal A (k) in the current frame k.

Here, the function r (k) may be defined as in Equation 10 below, and the valence function V (k) may be represented as in Equation 11 below.

[Equation 10]

[Equation 11]

는 작은 값이 되고 예상된 감정은 부정적인 경향이 된다.Here, r (k) may mean that the negativity of the expected feeling is determined by the amount of emotional adaptation in the situation. In other words, if the participant (subject) wants to avoid the situation,

Is small and the expected emotion tends to be negative.

에 의해 세밀하게 결정될 수 있으며, 함수 A(k)와 유사하게 함수 V(k)는 동일한 이동 평균 필터로 평활화될 수 있다. 여기서,

는 각각 r(k)와 V(k)의 가중된 평균을 의미할 수 있다.Based on this function, the valence value V (k) is the synchronous component

Can be determined in detail, and similarly to function A (k), function V (k) can be smoothed with the same moving average filter. here,

May mean a weighted average of r (k) and V (k), respectively.

Expressing emotional situations

The present invention expresses the emotional situation for the 2D emotion space from the valence value and the arousal value learned by the above calculation. The present invention uses two emotion values that fit the Gaussian process regression (GPR) model to approximate the Gaussian process regression (GPR) model. Here, the GPR model may be a nonparametric kernel based probabilistic model using a linear basic function and an accurate fitting method. This can generate an emotional curve as a representation of the emotional trajectory according to the situation as perceived by the human being.

device  Configuration

The present invention constructs a wearable device for the framework of the present invention and allows the user to act freely in everyday situations. The device collects the emotions of the user. Here, the device may collect the EEG signal of the user, and may capture various effects around the user by using a multi-modal sensor, for example, a camera, a physiological sensor.

Emotional situation Data set

로 정의될 수 있다. 여기서,

이고, Ns는 정서적 상황의 수를 의미하고, Ne는 상황에서 감정적 포인트의 쌍(valence, arousal)의 수를 의미할 수 있다.Emotional situation

It can be defined as. here,

, Ns may represent the number of emotional situations, and Ne may mean the number of emotional (valence, arousal) pairs in the situation.

의 곱에 의하여 결정될 수 있다. 매 30초로부터 모든 감정 포인트들은 정서적 라벨의 단일 집합을 생성하기 위하여 평균될 수 있다. 모든 상황의 지속기간은 세 주석자(annotators)에 의해 매뉴얼적으로 결정될 수 있다. 어떤 상황은 실측 값(ground truth)으로 SAM-등급 라벨을 가질 수 있다. SAM-등급 상황은

로 정의될 수 있으며,

는 Ns 내에서 SAM-등급 상황의 인덱스를 의미할 수 있다. SAM-등급 상황은 본 발명의 프레임워크의 파라미터들을 추정하고 그 성능을 평가하는데 사용된다. 각 상황은 두 채널 EEG 신호와 가속도 측정 데이터를 포함한다. Ne is the frame rate of the camera and the duration of the situation in seconds.

It can be determined by the product of. From every 30 seconds all emotion points can be averaged to produce a single set of emotional labels. The duration of any situation can be determined manually by three annotators. Some situations may have a SAM-grade label as ground truth. SAM-grade situation

Can be defined as

May refer to the index of the SAM-class situation within Ns. The SAM-grade situation is used to estimate the parameters of the framework of the present invention and to evaluate its performance. Each situation includes two channel EEG signals and acceleration measurement data.

Parameter setting

의 최소 값을 가지도록 2와 -1로 설정될 수 있다.

와

는 매 오일 동안 최대 값에 의해 결정될 수 있다.To model the emotional situation and express it in an emotional curve, the parameters for each participant are set through a five-fold cross-validation scheme. These parameters are set to -1 / by the adaptive component in the first frame of the situation.

It can be set to 2 and -1 to have a minimum value of.

Wow

Can be determined by the maximum value during each oil.

,

과

에 대하여 SAM과 본 발명의 프레임워크 간 valence와 arousal 등급 거리에 관한 5배 교차 유효성 스킴 결과를 나타낸 것이다. 정서 커브는 공간과 시간적 상황에서 정서적 역학(affective dynamics)을 표현하고 정서적 라벨의 복수 쌍을 포함하기 때문에 SAM 등급의 쌍과 비교할 때 상이한 것을 알 수 있다. 이는 valence와 arousal 등급이 동일한 상황에서 감정을 표현하는데 불연속 값을 가지기 때문이다. 비교를 위해, 본 발명은 모든 참가자들에 대해 0부터 6까지 스케일된 정서적 라벨 쌍의 평균 제곱 오차 제곱근(RMSE; root mean squared errors)을 계산한다. 여기서, 정서적 라벨 쌍의 평균 제곱 오차 제곱근은 아래 <수학식 12>와 같이 계산될 수 있다.5 shows different parameters

,

and

The results show a 5-fold cross-validation scheme for valence and arousal rating distances between SAM and the framework of the present invention. It can be seen that the emotional curves are different when compared to pairs of SAM grades because they represent affective dynamics in spatial and temporal situations and include multiple pairs of emotional labels. This is because valence and arousal grades have discrete values for expressing emotions in the same situation. For comparison, the present invention calculates root mean squared errors (RMSE) of emotional label pairs scaled from 0 to 6 for all participants. Here, the square root of the mean square error of the emotional label pair may be calculated as shown in Equation 12 below.

[Equation 12]

는 예측된 정서적 라벨의 쌍을 의미하고,

는 상황

에서 실측 값 라벨의 쌍을 의미할 수 있다. 0은 부정적 밸런스 등급을 표현하고, 3은 중립 밸런스 등급을 표현하며, 6은 긍정적 밸런스 등급을 표현한다. 또한, 0은 중립 어라우절 등급을 표현하며, 3은 낮은 어라우절 등급을 표현하며, 6은 높은 어라우절 등급을 표현한다.here,

Means a pair of predicted emotional labels,

Situation

May mean a pair of measured value labels. 0 represents a negative balance grade, 3 represents a neutral balance grade, and 6 represents a positive balance grade. In addition, 0 represents a neutral augmentation grade, 3 represents a low augmentation grade, and 6 represents a high augmentation grade.

와

가 어라우절 값 A(k)를 결정하는 반면, 밸런스 값 V(k)는

와

파라미터에 의해 결정된다. 파라미터

은 적응 구성 요소 l(k)가 0이 되는 경우 결정된다. 이는 밸런스 값 V(k)의 부호와 어라우절 값 A(K)의 증가에 직접적으로 영향을 준다.

가 더 작은 값을 가지게 되면, 적응 구성 요소 l(k)는 0이 되고 밸런스 값 V(k)의 부호는 앞선 k에서 양이 된다. 도 7a에 도시된 바와 같이, 밸런스 등급에 대한 최고 성능은 0.5이고, 두 개의 서로 다른 포인트 0.25와 0.75은 밸런스 등급에 대하여 그 다음 최고 파라미터이다. 반대로, 도 7b에 도시된 바와 같이 어라우절 등급의 거리는 대부분의 참가자에 대해 0.5 이후에 최소화된다. 파라미터

과

는 모션 구성 요소와 적응 구성 요소에 관한 어라우절 레벨을 결정한다. 그 결과, 본 발명의 시스템은 도 7c에 도시된 바와 같이 파라미터

에 대하여 대부분의 참가자에 대해 0.7과 0.8 사이에서 어라우절 등급에 대하여 최고의 성능을 가지는 것을 알 수 있다.

이 0.85보다 크고 0.4보다 작은 경우 범위를 벗어난 거리는 거의 동일하게 유지된다. 파라미터

과

는 밸런스 레벨을 결정한다.

이 더 큰 값을 가지게 되면 그 결과 밸런스는 접근-회피 행동과 연관된 감정을 표현할 수 없다. 도 7d에 도시된 바와 같이, 본 발명에 따른 시스템의 성능은 파라미터

이 변하는 경우 현저하게 변동하는 것을 알 수 있다. 본 발명에서 모든 참가자에 대해

과

을 0.75와 0.5로 설정하고,

은 실험에서 각 개인에 대해 최소 거리를 만들기 위하여 0.25, 0.5, 0.75에서 선택될 수 있다.

Wow

Determines the augment value A (k), while the balance value V (k)

Wow

Determined by the parameter. parameter

Is determined when the adaptive component l (k) becomes zero. This directly affects the sign of the balance value V (k) and the increase in the augment value A (K).

Has a smaller value, the adaptive component l (k) becomes zero and the sign of the balance value V (k) becomes positive at the preceding k. As shown in FIG. 7A, the best performance for the balance class is 0.5, and two different points 0.25 and 0.75 are the next best parameters for the balance class. In contrast, as shown in FIG. 7B, the distance of the augmentation rating is minimized after 0.5 for most participants. parameter

and

Determines the augmentation level for the motion component and the adaptive component. As a result, the system of the present invention has a parameter as shown in Fig. 7C.

We can see that we have the best performance for the augmentation grade between 0.7 and 0.8 for most participants.

If this is greater than 0.85 and less than 0.4, the out-of-range distance remains nearly the same. parameter

and

Determines the balance level.

With this larger value, the balance cannot express the emotions associated with approach-avoidance behavior. As shown in Fig. 7d, the performance of the system according to the invention is a parameter.

It can be seen that if this changes significantly. For all participants in the present invention

and

Is set to 0.75 and 0.5,

Can be chosen from 0.25, 0.5, 0.75 to make the minimum distance for each individual in the experiment.

EEG data preprocessing and setup for classification

In the preprocessing process, the same blind source separation technique is used to remove eye artifacts and a highpass filter with a 2Hz cutoff frequency using the EEGlab toolbox. A constrained independent component analysis (cICA) algorithm is applied to refine the signal by eliminating motion artifacts. The cICA algorithm is an extended algorithm of ICA.

EEG signals are vulnerable to motion artifacts. The present invention can improve the quality of the EEG signal by abandoning or eliminating the EEG signal highly correlated with the motion artifact, instead of separating and eliminating the motion artifact from the EEG signal caused by body movement. To this end, the present invention can divide the EEG signal into two separate groups by changing the accelerometer data G () in the range from 0 to 1. From each of the two separate groups, features such as average power, maximum amplitude, standard deviation of amplitude, amplitude kurtosis and amplitude skewness are extracted. The extracted features are then matrixed to describe the main features of the EEG. After the features are expressed in two-dimensional space using principal component analysis (PCA), the Battacharyya distance between two groups in two-dimensional space is calculated. Optical G () is a differentiator such as clean and contaminated EEG, determined by the maximum distance between the two groups. To analyze the physiological and psychological relationships in the present invention, EEG spectral power, three components and emotional labels are used in emotional situations where accelerometer data G () is less than optimal.

In order to study the efficiency of emotional labels obtained from emotional curves using EEG signals, the present invention is directed to the emotional curves in seven states, e.g., low-arousal / negative valence), low-arousal / neutral valence (LAUV), low-arousal / positive valence (LAPV), mid-arousal / negative valence, mid-arousal / positive valence (MAPV), high arousal / negative valence (HANV) and high-arousal balance (HAPV) / positive valence). Here, MANV and HAUV can be excluded because the present invention is not characterized in the actual experiment.

Some studies that extract EEG-based features from emotion recognition categorize into three domains: time, time frequency, and frequency. Assuming that the signal is fixed during the hearing period, the frequency domain feature is the most used of the three domains. Thus, the present invention uses frequency domain features in different frequency bands, for example, higher order spectra (HOS), power spectral density (PSD).

의 평균을 사용한다. 바이코히어런스는 신호 x(t)의 정규화된 바이스펙트럼

일 수 있다. 신호들은 겹치지 않는 세그먼트들로 분할되고, 각 세그먼트 내에서 데이터는 해닝(Hanning) 윈도우 및 푸리에 변환된다. 여기서, 바이스펙트럼은 아래 <수학식 13>과 같이 정의될 수 있다.HOS features are used to analyze human emotions as spectral representations of higher orders or cumulants of the signal. In particular, the present invention, in order to study the efficacy of the emotional label obtained from the emotional curve using the EEG signal, four frequency bands, for example, theta (4 ~ 7Hz), alpha (8 ~ 13Hz), beta (14 ~ 29Hz) , Bicoherence in the gamma (30-45 Hz) frequency band

Use the average of Bicoherence is the normalized bispectrum of the signal x (t)

Can be. The signals are divided into non-overlapping segments, within which data the Hanning window and the Fourier transform. Here, the bispectrum may be defined as in Equation 13 below.

[Equation 13]

는 신호 x(t)의 푸리에 변환을 의미하고,

는

의 복소 공액(complex conjugate)을 의미할 수 있다. 바이스펙트럼은 신호의 다른 구성 요소의 위상 정보를 보전한다. 두 주파수 구성 요소

과

는

의 주파수에서 세번째 구성 요소가 존재하는 경우 위상으로 결합된다. 바이코히어런스

는 아래 <수학식 14>와 같이 정의될 수 있다.here,

Means the Fourier transform of the signal x (t),

Is

It may mean a complex conjugate of. Bispectrum preserves the phase information of the other components of the signal. Two frequency components

and

Is

If there is a third component at the frequency of the phase is combined. Bicoherence

May be defined as in Equation 14 below.

[Equation 14]

는

에서 파워 스펙트럼을 의미할 수 있다. 바이코히어런스는 두 주파수 구성 요소 간의 위상 커플링 범위를 정량화한다. 결과 주파수 분해능은

과

축 모두에서 1Hz이다. 네 개의 주파수 대역에서 바이코히어런스

의 평균 크기는 아래 <수학식 15>와 같이 계산될 수 있다.here,

Is

Can mean the power spectrum. Bicoherence quantifies the range of phase coupling between two frequency components. The resulting frequency resolution

and

1 Hz on both axes. Bicoherence in Four Frequency Bands

The average size of can be calculated as shown in Equation 15 below.

[Equation 15]

는 두 주파수 대역 q1과 q2에서 주파수 구성 요소들을 의미할 수 있다. PSD의 파워 특징은 Welchs 방법을 이용하여 추정되고 네 개의 주파수 대역으로 나누어진다.

와 네 개 주파수 대역의 평균 파워는 EEG 신호를 이용한 정서 커브로부터 획득된 정석적 라벨의 상관을 분석하는데 사용된다.Here, q1 and q2 each mean a frequency band,

May mean frequency components in two frequency bands q1 and q2. The power characteristic of the PSD is estimated using the Welchs method and divided into four frequency bands.

The average power of the four and four frequency bands is used to analyze the correlation of the static label obtained from the emotional curve using the EEG signal.

FIG. 6 shows an example of an emotional curve combining an augmented curve and a balance curve. Each curve shown in FIG. 6 represents an emotional expression of an emotional situation in a participant's daily life, and an average curve of a parabolic shape is neutral. It covers the VA emotion space with the exception of some emotions characterized by balance and high level augmentation. Here, the dashed line represents the emotional curve for the participants, and the solid line represents the average curve of all the participants.

Physiological specificity : The ANOVA test results for bicoherence magnitude and PSD in four frequency bands of seven emotional states have a p value of 0.0679 in the beta frequency band and a p value lower than 0.05 in the remaining frequency bands. The p-value is due to the magnitude of the bicoherence in all frequency bands, while the PSDs at theta, alpha, and gamma frequencies differ significantly from the seven emotional states in the three frequency bands. These results indicate that PSD and bicoherence can be usefully used as physiological features to classify emotions.

Based on the results of ANOVA, the present invention can design a classification process that can better assess the effectiveness of an emotional curve to serve as a reliable emotional label in EEG-based emotion recognition. In particular, this name extracts PSD and Vicohererson features from four frequency bands, uses mutual information for feature selection, and has a 5th order polynomial kernel with a 5th order polynomial kernel that produces the highest accuracy in recognition. Classify the selected features into seven emotional states. The effectiveness of an emotional curve as a reliable emotional label to distinguish different emotional states can be assessed by thematic classification performance.

FIG. 7 shows an exemplary diagram of the results of the classification procedure with a 5-fold cross validity scheme for all participants, showing the average accuracy of all individual results for seven emotional states.

As shown in FIG. 7, the accuracy for recognizing seven emotional states is 66.65%, which means that the EEG signals grouped by the emotional labels have distinct patterns. The LAUV state has the lowest standard deviation of 3.7 and the highest accuracy, while the LAPV state and the LANV state have relatively higher standard deviations of 7.09 and 7.17 than the other five states. This means that participants have different activation patterns in the LAPV and LANV states, while having similar patterns of brain activity, such as feeling calm and relaxed in the LAUV state.

Physiological Characteristics : The present invention examines the statistical relationship between EEG spectral power and emotional labels in four frequency bands to reduce the gap between psychological measurements and physiological evidence. From Table 1 below, it can be seen that the alpha frequency components have a higher correlation for the two classes.

Here, F3 and F4 may refer to two electrodes for detecting the EEG signal.

The emotional dataset of the present invention includes hand movements of the participants, and since the Mu rhythm (8-11 Hz) activated by the movement in the motor cortical region is strongly related to the alpha frequency component, the activation of the motor cortex associated with movement is Correlated with the balance grade or the augmentation grade.

However, this does not mean that psychological measurements of hand movements are not associated with biological changes. To determine if the alpha frequency band is contaminated by hand movements and correlates with psychological measurements, the present invention calculates the power of four parking bands and the correlation coefficient between the three components. Table 2 below shows the correlation for each component between physiological brain activity and emotional state.

As described above, the motion component and the sync component are affected by the movement, but only the sync component reflects the user's approaching behavior in psychology and quantifies it in the balance class. The association between physiological changes in the alpha frequency band and psychological measurements of motion with the coefficient values in the alpha frequency band associated with the motion component and the synchronous component cannot be the result of motor cortical activation. This is because Mu rhythm is related to activation and positively related only to synchronous components, not motion components.

In addition, the present invention makes clear that emotional labels include physiological features attributable to psychological phenomena. To investigate the relationship between EEG spectral power and the three components, the present invention uses a Granger causality test. Here, the Granger causal test can analyze the flow of information between two series. If the pattern at x1 repeats at x2 within a certain time period, it can be said that the x2 Granger causes x1. At this time, the present invention may use a multivariate Granger causality test fitted by a vector autoregressive time series.

The present invention calculates Granger causality with EEG spectral power in four frequency bands from three components for all participants. It can be seen that the average of 0.29, 0.36 and 0.08 of the causal test have positive values for the EEG spectral power, which are under the influence of three components, respectively. The alpha and theta frequency bands have a relatively high causality and are affected by all components compared to other components. In contrast, the beta frequency band has low causality under both terms of the three components.

In other emotional labels, the characteristics of brain signals are analyzed by correlation and causal tests, and EEG-based statistical analysis shows that physiological responses correlate with continuous emotional labels.

As such, the framework according to the present invention performs emotional contextual expression and modeling, and may provide the following contributions.

(1) Emotional Situation Representation: The present invention presents an emotional curve. Emotion curves are cumulative sets of points over the balance and augmented emotional space over time, representing emotional dynamics in the real environment.

(2) Emotional Situation Modeling: The present invention senses the expected feeling and tracks changes in the situation when an image sequence is given. The present invention presents three components, modeling motivation, motion and adaptation, to model changes in the situation, which can be computed with low level visual features extracted to reflect the emotional response to the situation. In particular, the motivational component is based on the tendency to avoid access and quantifies the feeling of behavioral trends from emotional behaviors that occur in response to emotional content in the context.

(3) Physiological experiments to verify the effect of emotional labeling: The present invention evaluates using a series of long-term life records for several days in a real-world scenario, and investigates the characteristics of brain signals for different emotional labels. Analyze Statistical analysis based on electroencephalography (EEG) shows that physiological responses are consistently associated with emotional labels.

Such a framework according to the present invention may be implemented as an emotion estimation system. For example, the emotion estimation system according to an embodiment of the present invention may include a calculator, a modeler, a generator, and an estimator. Of course, the configuration means herein may vary depending on the situation.

The calculator calculates a motion component, a motivation component, and an adaptation component through learning of an emotional situation of the user using the image sequence.

In this case, the calculation unit may calculate the motion component by characterizing and quantifying a motion of an emotional object between adjacent frames using optical flow estimation for the image sequence.

In addition, the calculator may calculate a motion component by removing motion noise using motion activity estimated in each image frame of the image sequence and accelerometer data collected by an accelerometer attached to the user's body.

Furthermore, the calculator may acquire a protrusion area corresponding to the user's interest in the image frame of the image sequence, and calculate the synchronization component based on the acquired protrusion area.

Furthermore, the calculator may calculate the synchronization component by calculating the divergence and rotation using the optical flow of a certain area around the target of interest in the image frame of the image sequence.

Furthermore, the calculator may calculate the adaptive component based on the emotional situation length accumulated up to the current frame of the image sequence that changes over time.

The modeling unit models a balance model and an arousal model based on the calculated motion component, synchronization component, and adaptive component.

In this case, the modeling unit may model the Aargument model using the calculated motion component and the adaptive component, and may model the balance model using the Aargument model and the synchronization component.

The generation unit generates an emotional curve based on the emotional situation represented on the emotional space by the modeled balance model and the augmentation model.

The estimator estimates the emotion value of the user using the emotion curve.

In addition, the system according to another embodiment of the present invention models the balance model and the arousal model based on the user's behavior using the image sequence, and the emotion is modeled by the modeled balance model and the By generating an emotional curve based on the emotional situation represented in space, the generated emotional curve may be used to estimate the user's emotion value.

Here, the system may detect the user's interest from the image sequence, and model a balance model and an arousal model based on the interaction with the detected interest.

Although the description is omitted in the system of the present invention, it will be apparent to those skilled in the art that the system according to the present invention may include all the contents described above with reference to FIGS. 3 to 7.

The system or apparatus described above may be implemented with hardware components, software components, and / or combinations of hardware components and software components. For example, the systems, devices, and components described in the embodiments may include, for example, processors, controllers, arithmetic logic units (ALUs), digital signal processors, microcomputers, field programmable arrays (FPAs). ), A programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For convenience of explanation, one processing device may be described as being used, but one of ordinary skill in the art will appreciate that the processing device includes a plurality of processing elements and / or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as parallel processors.

The software may include a computer program, code, instructions, or a combination of one or more of the above, and configure the processing device to operate as desired, or process independently or collectively. You can command the device. Software and / or data may be any type of machine, component, physical device, virtual equipment, computer storage medium or device in order to be interpreted by or to provide instructions or data to the processing device. Or may be permanently or temporarily embodied in a signal wave to be transmitted. The software may be distributed over networked computer systems so that they may be stored or executed in a distributed manner. Software and data may be stored on one or more computer readable recording media.

The method according to the embodiments may be embodied in the form of program instructions that may be executed by various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the media may be those specially designed and constructed for the purposes of the embodiments, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks, such as floppy disks. Magneto-optical media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Although the embodiments have been described by the limited embodiments and the drawings as described above, various modifications and variations are possible to those skilled in the art from the above description. For example, the described techniques may be performed in a different order than the described method, and / or components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different form than the described method, or other components Or even if replaced or substituted by equivalents, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are within the scope of the claims that follow.

Claims (22)

Calculating a motion component, a motivation component, and an adaptation component through learning of an emotional situation of the user using the image sequence;
Modeling a balance model and an arousal model based on the calculated motion component, synchronization component, and adaptive component; And
Generating an emotional curve based on the emotional situation represented on the emotional space by the modeled balance model and the ʻauraud 'model
Emotion estimation method comprising a.
The method of claim 1,
The calculating step
For the image sequence, calculating the motion component by characterizing and quantifying the motion of an emotional object between adjacent frames using optical flow estimation.
The method of claim 1,
The calculating step
The motion component is calculated by removing motion noise using motion activity estimated in each image frame of the image sequence and accelerometer data collected by an accelerometer attached to the user's body. .
The method of claim 1,
The calculating step
And a protruding region corresponding to the user's interest in the image frame of the image sequence, and calculating the synchronization component based on the acquired protruding region.
The method of claim 4, wherein
The calculating step
And calculating the synchronization component by calculating divergence and rotation using an optical flow of a predetermined region around the target of interest in the image frame of the image sequence.
The method of claim 1,
The calculating step
And calculating the adaptive component based on the emotional context length accumulated up to the current frame of the image sequence that changes over time.
The method of claim 1,
The modeling step
Emotion estimation method, characterized in that for modeling the Augment model using the calculated motion component and adaptive component.
The method of claim 1,
The modeling step
Emotion estimation method characterized in that for modeling the balance model by using the augmentation model and the synchronization component.
Modeling a balance model and an arousal model based on the user's behavior using the image sequence;
Generating an emotional curve based on the emotional situation represented on the emotional space by the modeled balance model and the ʻauraud 'model; And
Estimating the emotion value of the user using the emotional curve
Emotion estimation method comprising a.
The method of claim 9,
The modeling step
Calculating a motion component, a motivation component and an adaptation component based on the user's behavior; And
Modeling a balance model and an arousal model based on the calculated motion component, synchronization component, and adaptive component
Emotion estimation method comprising a.
The method of claim 9,
The modeling step
And detecting the interest of the user from the image sequence and modeling the balance model and the arousal model based on the interaction with the detected interest.
A calculation unit configured to calculate a motion component, a motivation component, and an adaptation component through learning of an emotional situation of the user using an image sequence;
A modeling unit for modeling a balance model and an arousal model based on the calculated motion component, synchronization component, and adaptive component; And
A generation unit for generating an emotional curve based on the emotional situation expressed in the emotion space by the modeled balance model and the AU clause model
Emotion estimation system comprising a.
The method of claim 12,
The calculation unit
For said image sequence, calculating said motion component by characterizing and quantifying the motion of an emotional object between adjacent frames using optical flow estimation.
The method of claim 12,
The calculation unit
An emotion estimation system, wherein the motion component is calculated by removing motion noise using motion estimation estimated at each image frame of the image sequence and accelerometer data collected by an accelerometer attached to the user's body .
The method of claim 12,
The calculation unit
And a protruding region corresponding to the user's interest in the image frame of the image sequence, and calculating the synchronization component based on the acquired protruding region.
The method of claim 15,
The calculation unit
And calculating the synchronization component by calculating divergence and rotation using an optical flow of a predetermined region around the target of interest in the image frame of the image sequence.
The method of claim 12,
The calculation unit
Wherein the adaptive component is calculated based on the emotional context length accumulated up to the current frame of the image sequence that changes over time.
The method of claim 12,
The modeling unit
Emotion estimating system, characterized in that for modeling the Auraugraph model using the calculated motion component and adaptive component.
The method of claim 12,
The modeling unit
Emotion estimating system, characterized in that for modeling the balance model by using the augmentation model and the synchronization component.
A modeling unit for modeling a balance model and an arousal model based on a user's behavior using an image sequence;
A generator configured to generate an emotional curve based on the emotional situation represented in the emotion space by the modeled balance model and the ʻauraud 'model; And
An estimator estimating an emotion value of the user using the emotional curve
Emotion estimation system comprising a.
The method of claim 20,
The modeling unit
Calculate a motion component, a motivation component and an adaptation component based on the user's behavior, and based on the calculated motion component, the synchronization component and the adaptive component And modeling a balance model and an arousal model.
The method of claim 20,
The modeling unit
And a balance model and an arousal model based on interactions with the detected targets of interest.

Images