KR20180019417A - Reasoning Method and System of Empathic Emotion Based on Video Analysis - Google Patents

Abstract

The present invention relates to an empathic emotion analyzing method through a video analysis and a system thereof. A reasoning method comprises the following steps of: obtaining a moving picture from a subject placed under social environments; extracting video information with respect to a difference between before and after frames from the moving picture; detecting a change amount between frames from the video information to extract a motion data value of the subject; and analyzing a motion amount of the subject to determine whether the subject is sympathized.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and system for empathic emotion inference using image analysis,

The present invention relates to a method for inferring social sympathy emotion through video analysis, and more particularly, to a method for inferring empathy emotion based on motion information.

Empathy means to perceive another person's thoughts or psychological state by feeling as a person's position. This empathy is semantically different from sympathy. Compared to sympathy, empathy is more inclusive, involving behavioral elements and even the feeling of imagination (such experience) (Rogers, 1957; Basch, 1983). The more empathic the better, the better the emotional exchange with the other person, and the more social feeling such as sociality can be extended.

Human movements include various emotional information. Gestures, facial expressions, and so on. These emotional information can lead to emotional contagion to the members of the group, and confirm previous studies that reported feeling the same emotions as specific members (Non-Patent Document 3). These studies require more specific systems, and continuous research is required.

Basch, M. F. (1983). Empathic understanding: A review of the concept and some theoretical considerations. Journal of the American Psychoanalytic Association Rogers, C. R. (1957). The necessary and sufficient conditions of therapeutic personality change. Journal of consulting psychology, 21 (2), 95. Barsade, S. G. (2002). The Ripple Effect: Emotional Contagion and Its Influence on Group Behavior. Administrative Science Quarterly, 47 (4). Pupil size synchronization difference according to intimacy between two people,

The present invention proposes a spatial emotion inference method and system using image analysis.

Space Sensitivity Reasoning Method According to the Present Invention:

Acquiring a moving picture from a subject;

Extracting image information about a difference between before and after frames from the moving image;

Detecting a change amount between frames from the image information and extracting a motion amount of the subject from the detected amount;

And analyzing the motion information to determine whether the subject is sympathetic to the other person.

According to an embodiment of the present invention, the amount of change between the frames is obtained from the difference between the RGB values of the image pixels per frame.

According to an exemplary embodiment of the present invention, the inter-frame variation due to the difference in RGB values is obtained by calculating the RGB difference values between adjacent frames, and the difference value is calculated based on the average value, , The motion amount can be optimized by continuously accumulating the difference of the smallest motion as the motion data value.

According to one embodiment of the present invention, the determination of empathy of the subject is performed by extracting a mean value and a standard deviation (SD) from a motion data value (x) in a predetermined period (T1) To obtain the Z score.

Z = (x-mean) / SD

According to an embodiment of the present invention, a Z value calculated in units of the period (T1) is calculated with respect to a motion amount per second based on a predetermined window size and an interval time.

Here, an average (Zm) of the Z scores during the period (T1) is obtained, and the average (Zm) is applied as a reference value or a threshold value for determining whether or not the subject agrees.

An empirical reasoning system using image analysis to perform the above method:

A moving picture camera for acquiring the moving picture;

An image processing unit for processing moving pictures from the moving picture camera;

An analysis unit for analyzing the image from the image processing unit to infer the empathy emotions of the subject; And

And a display for displaying an analysis result from the analysis unit.

1 is an overall schematic flow diagram of an emotional analysis method according to the present invention.
FIG. 2 shows the process from the image acquisition to the empathy determination in more detail by further segmenting the process of FIG.
FIG. 3A illustrates an experiment setting in the empathy condition between a leader and a follower in a method implemented to verify the empathic analysis method according to the present invention.
FIG. 3B illustrates an experiment setting in a non-empathy condition between a reader and a follower in a method performed to verify the empathic analysis method according to the present invention.
4 is a diagram illustrating a configuration of a sympathetic emotional analysis system according to the present invention.
FIG. 5 is a graph showing experimental results performed to verify the present invention. FIG.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an embodiment of a method for analyzing empathy through image analysis according to the present invention and a system for applying the same will be described with reference to the accompanying drawings.

1 is an overall schematic flow diagram of an emotional analysis method according to the present invention.

The whole process of the method according to the invention is shown. That is, the present invention includes a step of capturing a moving image from two subjects in a face-to-face relationship with a crowd or a 1: 1 person, collecting image data (S1), pre-processing the collected moving image (S2) And a data analysis step (S3) of judging whether the subject is sympathetic or not.

FIG. 2 shows the process from the image acquisition to the empathy determination in more detail by further segmenting the process of FIG.

The flow on the left side of FIG. 2 shows a process of extracting motion data, and a process of determining whether or not the right side is sympathetic.

First, a process of extracting motion data from an image obtained from two subjects will be described with reference to the left side of FIG.

(S11) In order to extract the amount of motion of the two subjects (A, B), the subject's body, preferably the head or face, is photographed, and the image of the predetermined resolution is received in real time. Here, the amount of change per frame is extracted. Here, the amount of change per frame includes image information about the difference between adjacent front and rear frames, for example, a current frame and a previous frame And extracted.

(S12) Each input image of the subject is a size corresponding to the resolution, and R, G, and B values for each pixel are extracted on a frame-by-frame basis. According to an embodiment of the present invention, the image information on the frame difference is the difference between the R, G, and B pixel values between frames.

(S13) The change or variation information of each pixel (R, G, B) value should include changed information of the R, G, B pixel values of the current frame in comparison with the previous frame. , G, and B pixels (values). The formula is as follows.

This process is also performed for each of the images of the two subjects, thereby obtaining the rate of change of all the pixels of the current frame as compared to the previous frame.

 (S14) The pixel variation information of the two subjects extracted in the above process includes the motion change information in the image. The normalization process is a process for calculating the change information in which the motion change information is continuously calculated within the same range. The normalization equation for each of the R, G, and B pixels is as follows. This normalization is performed separately for R, G, and B pixels.

The Max and Min values above represent the highest and lowest pixel values in a frame.

After obtaining the motion value through the above process, it is judged whether or not the subject is sympathetic through the following process.

(S15) The Output Movement Data process is a task for weighting information that is continuously changed through the cumulative operation of the extracted information, and removing motion that is not. The formula for the Accumulated Normalization Value (ANV) is as follows and is calculated for each of the R, G, and B pixels.

Here, Current Normalized Pixels refers to the pixels of the current frame, and Previous Normalized Pixels refers to pixels of the previous frame.

After the accumulation operation, the motion data (Movement Data, MD) is converted to a total average (Grand Average).

Here, n = image resolution (width x height), e.g., n = 640x480.

(S16) The motion value is input through the above process and stored in the buffer.

(S17) The motion data for a predetermined time, for example, 30 seconds, is loaded from the buffer.

In order to adjust the range of motion information input for each image of two subjects A and B, the motion range of each image is adjusted through the following equation of Normalization of Movement Data (Z, Z-score).

Here, X represents a motion signal of a specified buffer size, m represents a motion average value for a specified buffer size, and SD represents a standard deviation value for a specified buffer size.

A mean value and a standard deviation (SD value) are extracted by using a motion value (x) for a predetermined period (T1), for example, 30 seconds, and the amount of change in motion is obtained by the above equation.

(S17) The Z score computed every 30 seconds through the above equation can be used to determine a window size, e.g., 3 seconds, a predetermined interval time, e.g., every second Calculate the value for the amount. Here, a Z-score average (Zm) for a predetermined period, i.e., 30 seconds, is obtained.

The SD value of the measured Zm can be used to calculate the sympathetic score. The sympathetic score system can grasp the degree of sympathy by considering consecutive scoring system based on the lowest value and the highest value. For example, the score can be systematized using the correction value based on the SD average.

(S18) As a process of determining the level of empathy, the standard deviation value is extracted from the extracted Z value, and the score is reflected in the empathy degree as it approaches the reference value shown in FIG.

In order to evaluate the accuracy of the method of the present invention as described above, the following experiment was conducted.

FIG. 3A illustrates an experiment setting in the empathy condition between a leader and a follower in a method implemented to verify the empathic analysis method according to the present invention.

FIG. 3B illustrates an experiment setting in a non-empathy condition between a reader and a follower in a method performed to verify the empathic analysis method according to the present invention.

According to the experiment based on the action response to the visual stimulus for a human monkey, the cognitive process processing time from the visual information to the process of taking the action by feedback as the feedback is at least 180 ~ 260 ms (Thorpe, 2001). In addition, using the facial photographs, the highest discrimination rate was obtained at 1 to 3 seconds within a total of 15 seconds for a given facial discrimination result by MRI in the task of discriminating according to race. Accordingly, the image data analysis was performed by setting the time taken to imitate the facial expression to a maximum of 3 seconds, and applying a window size of 3 seconds and a time interval of 1 second.

The data on the empathic analysis were used to evaluate the empathy of the two people. A total of 34 participants, 17 pairs, defined empathy as sharing emotions by understanding emotions of others. Subjects in each group set up conditions with sympathy through interactions that mimic facial expressions by acting as a leader and follower. We set up the condition that there is no sympathy to mimic the expression presented on the screen without interaction. The facial imitating task was performed for about 4 minutes. During the facial imitation test, 640 * 480, 30 fps was set around the upper body of the subject's face.

As shown in Fig. 4, the system includes a moving picture camera 10 for photographing an upper body or a face of a follower 1b that imitates the facial expression of a subject or a reader 1a, an image processing unit An analysis unit 30 for analyzing the subject's social empathy state, and a display 40 for displaying an analysis result of the subject empathy state. Such a system can be implemented as a computer-based general purpose apparatus or a dedicated apparatus having a moving picture camera.

The results of analysis of the data obtained from 34 subjects as described above are as shown in FIG. According to the results, SD was found to be smaller when the empathy was judged (z = -2.157, p = .031).

In this experiment, the face of the subject was photographed and the Z score of the face motion was analyzed. The mean (mean ₍₃₀₎ and variance (SD ₍₃₀₎ ) values for 30 seconds in the 4 minutes image were obtained and the window size was calculated by applying an interval time of 3 seconds and 1 second The Z-score average of reader and follower was extracted.

The following is a formula to calculate the Z-score (Z _leader , Z _follower ) of the leader and follower for 30 seconds each.

Z _leader = (X-mean ₍₃₀₎ ) / SD ₍₃₀₎

Z _follower = (X-mean ₍₃₀₎ ) / SD ₍₃₀₎

The SD values of the leader and follower of empathy and non - empathy were calculated and compared.

The respondent sample t-test was conducted for two conditions that were sympathetic and not sympathetic. As a result, it was confirmed that the Z score of the non-empathy was significantly higher than that of the empathy (t (-2.287), p <0.035). The significance of the Z score means that the degree of variance of the facial expression movements of the two persons is large, which means that the expressions of the two persons do not move similarly. Therefore, it can be confirmed that the empathy is less in the empathy group than in the empathy group.

The present invention can easily determine the empathy formation in the social environment to which a subject or a specific person belongs by acquiring and analyzing the amount of fine movement change of a subject or a specific person.

The SD value thus measured can be used to calculate the sympathetic score.

The sympathetic score system can grasp the degree of sympathy by considering consecutive scoring system based on the lowest value and the highest value. For example, the score can be systematized using the correction value based on the SD average.

Correction value = 50 / average SD value

Sympathetic score = 100- (SD value * correction value), based on 100 points

While this invention has been described in connection with what is presently considered to be the most practical and preferred embodiment, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the invention. Therefore, the true scope of protection of the present invention should be defined only by the appended claims.

1a: Reader 1b: Followers
10: video camera 20: image processing unit
30: Analysis section 40: Display

Claims (9)

Acquiring a moving picture from two subjects lying in a social environment;
Extracting image information about a difference between before and after frames from the moving image;
Detecting a rate of change between frames from the image information and extracting a motion data value (x) of the subject from the change rate;
Extracting a mean value and a standard deviation (SD) from the motion data value (x) in a predetermined period (T1) and obtaining a Z score (Z) using the following equation; And
And determining whether or not the two subjects are sympathetic or non-sympathetic according to the Z score.
Z = (x-mean) / SD The method according to claim 1,
Wherein the rate of change between frames is obtained from Variation of R, G, and B values of image pixels per frame. 3. The method of claim 2,
Wherein the variance of the R, G, and B values satisfies the following equation.

The method of claim 3,
Wherein the variance of the R, G and B values is normalized by the following equation.

5. The method according to any one of claims 1 to 4,
Wherein a value of a motion amount of two subjects every second is calculated based on a predetermined window size and an interval time of Z scores calculated in units of the period (T1) Empathy emotional reasoning method. 5. The method according to any one of claims 1 to 4,
Wherein one data (Convolution Data) is obtained through a linear sum of the motion data obtained from the images of the two subjects, and a standard deviation serving as a criterion for determining the empathy degree is obtained therefrom. 5. A system for performing the method according to any one of claims 1 to 4,
A moving picture camera for acquiring the moving picture;
An image processing unit for processing moving images from the moving picture camera;
An analysis unit for analyzing the image from the image processing unit to infer the empathy emotions of the subject; And
And a display for displaying an analysis result from the analysis unit. The method according to claim 6,
The analysis unit:
Wherein a value for each motion amount is calculated on the basis of a window size and an interval time of Z scores calculated in units of the period (T1). 8. The method of claim 7,
The analysis unit:
Wherein one data (Convolution Data) is obtained through a linear sum of the motion data obtained from the images of the two subjects, and a standard deviation serving as a criterion for determining empathy is obtained therefrom.

Images