KR102285998B1 - Method and apparatus for evaluating empathy for image contents - Google Patents

Abstract

The present invention provides a method and an apparatus for determining whether a viewer sympathizes with image content by a noncontact method. The determining method comprises: a step of showing image content including a character to a viewer through a display device; a step of photographing a face image of the viewer by a camera, and acquiring a face image of the character by using a capturing device; a step of extracting face micromotion data of the character and the viewer from the two face images through an image analyzer; a step of evaluating the similarity of the face micromotion data of the character and the viewer by a micromotion data analyzer; and a step of using a sympathy determining device to compare the similarity with an arbitrarily-set reference value to detect face micromotion synchronization of the viewer with the character to determine whether the viewer sympathizes with the image content.

Description

Method and apparatus for evaluating empathy for image contents

The present disclosure relates to a method and apparatus for evaluating empathy for video content, and to a method and apparatus for evaluating a viewer's empathy for video content based on micro-motion.

In the commercial media environment, competition to induce interest and participation of video content consumers, ie, viewers, for video content is very fierce.

The empathy of viewers who responds to the content increases their immersion in the content and increases their understanding of the intrinsic message they are trying to convey. Therefore, it is important to gain sympathy for video content from viewers every week.

The higher the viewer's sympathy for the video content, the more deeply moved by the embedded message, the more positive the attitude of the video content consumer.

In the case of commercial video advertisement content, it is possible to increase the will to purchase the product through a deep understanding and empathy for the brand and product delivered through the video content by increasing viewers' interest and empathy.

Consumers' high level of empathy for video advertising content deepens their understanding of advertising products and their value. It can be used as a rating guide to improve a media product, brand, or content.

Existing methods for measuring empathy for content include a subjective method and an objective method.

The subjective method mainly relies on subjective questionnaires about viewers exposed to video content. However, the evaluation of empathy by such a subjective questionnaire is inaccurate and has low reproducibility. And as a measurement method using objective and quantitative data, physiological information obtained from a viewer by a physical device is used. However, since this objective method relies on a physical device for acquiring physiological information, it imposes a psychological burden on the viewer as the subject.

Park, S., Choi, S. J., Mun, S., Whang, M. (2019). Measurement of emotional contagion using synchronization of heart rhythm pattern between two persons: Application to sales managers and sales force synchronization. Physiology and behavior, 200, 148-158. Lee Dong-won, Park Sang-in, Moon Seong-cheol, Hwang Min-cheol. (2019). A method for empathic recognition using cardiac response synchronization. Emotional Science, 22(1), 45-54. Balakrishnan, G., Durand, F., Guttag, J. (2013). Detecting pulse from head motions in video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3430-3437).

The present disclosure provides a method for evaluating empathy for video content by a non-contact method and an apparatus for applying the same.

The present disclosure provides a method for evaluating empathy for video content using facial micro-movement similarity analysis and an apparatus for applying the same.

According to one or more embodiments according to the present disclosure

presenting the video content including the characters to the viewer through a display device;

obtaining a facial image of the viewer viewing the video content using a camera;

extracting facial micro-movement information of characters and viewers from the image content and facial images through an image analyzer;

evaluating the degree of similarity between the facial micro-movement information of the person and the viewer by a micro-movement information analyzer; And

Comparing the degree of similarity to an arbitrarily set reference value using an emotion determiner, detecting the synchronization of the viewer's facial fine movements with respect to the character to determine whether the viewer sympathizes with the video content; empathy using facial micro-movement similarity analysis, including A judgment method is presented.

According to another embodiment of the present disclosure, the facial micro-movement data: may be extracted from a movement of a feature extracted from the facial image.

According to another embodiment of the present disclosure, the movement of the feature may be extracted from the coordinate movement of the feature between front and rear frames of the facial image.

According to another embodiment of the present disclosure, the facial micro-movement data may be extracted from a movement in the longitudinal direction (y-axis) among coordinate movements of the feature.

According to another embodiment of the present disclosure, the facial micro-movement data may be selected by filtering in a specific frequency range corresponding to a heartbeat.

According to another embodiment of the present disclosure, the facial micro-movement data may be processed by a band-pass filter of 0.5 to 5 Hz by frequency analysis.

According to another embodiment of the present disclosure, the similarity may be obtained from periodicity of facial micro-movement data of the character and the viewer.

According to one or more embodiments according to the present disclosure,

a display device for presenting image content;

a display device for displaying image content including characters;

a camera for acquiring a facial image of the viewer viewing the video content;

an image analyzer for extracting facial micro-movement information of characters and viewers from the image content and facial images;

a fine motion information analyzer for evaluating the degree of similarity between the facial fine motion information of the person and the viewer; And

An empathy determination device comprising a; empathy determination device for determining whether the viewer sympathizes with the video content by detecting the synchronization of the viewer's facial fine movement with respect to the character by comparing the similarity to an arbitrarily set reference value.

According to another embodiment of the present disclosure, the empathy determination apparatus may include a computer-based empathy determination system capable of image display, image analysis, analysis of fine movement information, and emotion determination.

According to another embodiment of the present disclosure, the image analyzer may extract facial micro-movement data from a movement of a feature extracted from the facial image.

According to another embodiment of the present disclosure, the image analyzer may extract the movement of the feature from the coordinate movement of the feature between front and rear frames of the facial image.

According to another embodiment of the present disclosure, the image analyzer may extract from the movement in the longitudinal direction (y-axis) among the coordinate movement of the feature.

According to another embodiment of the present disclosure, the image analyzer: may select the facial fine movement data in the range of 0.75 to 2.5 Hz by frequency analysis of the facial fine movement data.

According to another embodiment of the present disclosure, the emotion determiner may extract the similarity from periodicity of facial micro-movement data of the character and the viewer.

Content empathy analysis according to the present disclosure determines the degree of empathy between a person appearing in the content and a viewer who views the content using a camera image-based analysis method.

In this method, sympathy is determined by measuring the synchronization between the facial fine movement information extracted from the facial region of a person in the video content and the facial fine movement information extracted by photographing the facial region of the user watching the image with a camera. Facial micro-movement defined in the embodiment according to the present disclosure refers to a minute movement of the face, in which periodic vibrations caused by heartbeat generate periodic minute movements in the face along the carotid artery of a person. The method proposed in the present disclosure is practical because camera-based non-contact empathy determination is possible without attaching a separate sensor for measuring biosignals.

1 illustrates an overall structure of an empathy evaluation apparatus according to an embodiment of the present disclosure.
2 is a block diagram showing functional components of an empathy evaluation apparatus according to an embodiment of the present disclosure.
3 is a schematic flowchart of a method for evaluating empathy according to an embodiment of the present disclosure.
4 is a detailed flowchart of a process of extracting fine movement data according to an embodiment of the present disclosure.
5 shows a detailed flow from facial image acquisition to motion data extraction according to an embodiment of the present disclosure.
6 is a flowchart for signal inference and classification empathy level extraction according to an embodiment of the present disclosure. And
7 logically shows the process of evaluating the empathy level over three steps according to an embodiment of the present disclosure.

Hereinafter, a content empathy evaluation method and apparatus according to the present disclosure will be described with reference to the accompanying drawings.

1 schematically illustrates an empathy evaluation apparatus according to an embodiment of the present disclosure. The empathy evaluation device acquires a facial image of the viewer 20 while showing a specific video content to a subject or viewer 20 and evaluates whether the facial image is synchronized with the video content.

Specifically, when describing the empathy evaluation device, the hardware structure based on the computer system 1 including the input/output devices such as the keyboard 14, the mouse 15, the monitor 12, and the main body 11 to which they are connected have

The hardware may be implemented by a dedicated system according to another embodiment of the present disclosure. The technical scope of this empathy evaluation device is not limited by a specific hardware system.

More specifically, the empathy evaluation apparatus or system according to the present disclosure for performing the empathy evaluation method according to the present disclosure includes a moving picture camera 13 for photographing the subject 20's face, for example, a so-called web cam or web camera. can do. In addition, such a system processes image content and a face image obtained from a subject viewing the image content through an analysis algorithm provided in the form of software, and evaluates whether the viewer 20 sympathizes with the image content using this.

2 is a block diagram showing functional components of the empathy evaluation device.

As shown in FIG. 2 , the display device 12 and the camera 13 face the viewer 20 . The display device 12 is a device that visually presents the image content 114 to the viewer 20 . The camera 13 captures the face of the viewer 20 watching the video content 114 and converts the facial expression reacting to the video content 114 into video data.

The computer-based empathy evaluation system 1, in which the camera 13 and the display device 12 are connected, transmits the image player 115 that reproduces the image content 114 and the image player 115 to the display device 12 It includes an image capturer 116 that captures the image to be used in units of frames. The content image captured by the image capturer 116 is image data for comparison with the facial image of the viewer 20 captured by the camera 13 .

The video content 114 includes a character to which the viewer 20 can emotionally respond, and facial movements of the character are extracted.

The image analyzer 111 analyzes the facial image of the person and the facial image of the viewer, which are the captured images, and extracts fine movement data of both. The fine motion data has a fine motion component related to the heart data.

The fine motion analyzer 112 analyzes the fine motion data to analyze whether the viewer 20 emotionally sympathizes with the emotion expressed by the character.

The empathy evaluator 113 determines whether the user empathizes with the emotion expressed by the video content or the character by using the result from the micro-movement analysis 112 .

Above, the image analyzer, the fine movement analyzer, and the empathy evaluator may be provided by an analysis algorithm provided in the form of software.

3 is a schematic flowchart of a method for evaluating empathy according to an embodiment of the present disclosure.

S1: Facial Video Acquisition

Step S1 is a step of acquiring a facial image. Acquisition of the facial image includes shooting of the viewer's face by a camera and continuous capture of image content, as described above. At this stage, for example, two facial images at 30 fps are acquired.

S2: Signal Extraction

Step S2 is a process for extracting facial micro-movement signals including components due to cardiac response from two face images.

S3: Principal Component Analysis

Step S3 is a process for extracting a single facial fine motion signal having the same component from the facial fine motion signal extracted in the entire process, and a plurality of components are extracted through principal component analysis.

S4: Similarity Calculation

Step S4 is a process for calculating the similarity of the two extracted facial fine movement signals. After finding the Euclidian distance for each buffer (window), the average value is calculated.

S5: Classification Empathy Levels

In step S5, the empathy level is classified according to the subjective empathy score based on the similarity distance value calculated in the above step. Specifically, if the empathy evaluation score is less than the mean - standard deviation value, the two similarity distance values are classified into a group with a low empathy level, and if the empathy evaluation score is greater than the mean + standard deviation value, the two similarity distance values are divided into the empathy level. classified into this high group.

4 is a detailed flowchart for the process of extracting fine motion data in the above process, and FIG. 5 shows a specific flow from facial image acquisition to motion data extraction.

S21: Face Detection

In step S21, a face region is detected from the photographed and captured image. For example, a face may be detected using the Viola-Jones algorithm that uses the contrast feature that appears for each part of a person's face from a facial video.

S22: Area Selection

In step S22, as a process of selecting a specific region for extracting fine movement data, for example, a forehead and a nose region in which noise due to facial expression appears the least from a detected face region is selected.

S23: Feature Extraction

In step S23, a plurality of feature points for tracking are extracted from the selected area in comparison with other points. A Good-Feature-To-Track (GFTT) algorithm and a Facial Landmark Detection (FLD) algorithm may be applied to extracting these feature points. In this embodiment, 80 feature points are extracted by applying the GFTT algorithm.

S24: Feature Tracking

In step S24, motion data of each feature point extracted in step S23 is obtained. A Kanade-Lucas-Tomasi (KLT) tracking algorithm, a TM (Transformation Matrix)-based tracking algorithm, or the like may be applied to such feature tracking. In this embodiment, a facial fine movement signal is extracted by tracking the value of the movement of the y-coordinate value in the current frame compared to the previous frame for each feature point using the KLT algorithm for successive frames. A sliding window technique may be used to extract the facial fine movement signal, and the window size may be set to 30s and the interval size to 1s.

S25: Datal Extraction

In the data extraction step, facial fine movement signals (data) as many as the number of feature points extracted in the feature extraction step are obtained as a result of the sliding window technique.

6 is a flowchart for signal inference and classification empathy level extraction, and FIG. 7 logically shows the process of empathy level evaluation spanning three steps.

The Signal Estimation step is a process for extracting only a component due to a cardiac response from the extracted facial micro-motion signal or data.

S31: Band Pass Filter

In step S31, a frequency component corresponding to a heartbeat band is extracted from each facial fine movement signal. For this, only the frequency band corresponding to the heartbeat band is extracted using, for example, Butterworth Bandapss Filter (5 order, 0.75-2.5Hz).

S32: Principal Component Analysis

Step S32 is a process for extracting one facial fine motion signal having the same component from the facial fine motion signal extracted from each feature point, and a dimensionality reduction method is applied. Principle component analysis (PCA), independent component analysis (ICA), support vector decomposition (SVD), etc. may be applied to the dimension reduction method. In this embodiment, five components are extracted through principal component analysis. Using the feature that biosignals have periodicity for each component, the component showing the highest periodicity is extracted as the final facial fine movement signal. Periodicity is calculated as follows.

Here, s is a time series signal, FFT is a Fourier analysis method for converting a time series signal into a frequency band, and PS is a power spectrum of a frequency band for the time series signal s.

where Max Power is the largest power value across the entire power spectrum

where Total Power is the sum of the entire power spectrum.

Finally, the periodicity of the time series signal s is calculated as follows.

The viewer's sympathy for video content is judged through a three-step process as follows.

S41: Similarity Calculation

Step S41 is a process for calculating the similarity for the two extracted facial fine movement signals (content 1, user 1). Each buffer (window, W ₀ ~ W _n ) is the Euclidean distance (EUC). _d ) and then calculate the average value (Average).

S42: Classification Empathy Levels

In step S42, the level of empathy is classified according to the subjective empathy score based on the similarity distance value calculated in the above step.

S43: Determine Empathy

When the empathy score is determined, if the evaluated empathy score is less than the empathy score mean (m) - standard deviation (d) value, the two similarity distance values are set to the low level of empathy. If the empathy evaluation score is greater than the mean (m) + standard deviation (d) value, the two similarity distance values (EUCd) are classified as a high level of empathy.

In order to verify the content empathy evaluation according to an embodiment of the present disclosure, a normality test by statistical analysis was performed. In this step, statistically verifying that the similarity distance values of two facial fine movements classified into two levels of empathy according to the empathy score have a small similarity distance value in a group with a high empathy level and a large similarity distance value in a group with a low empathy level. am. At this stage, a group with a high level of empathy has a small similarity distance value, meaning that the group with a high level of empathy has a high degree of similarity between the two facial movements. The specific method of statistical verification is the Independent T-test, a two-level interlevel parameter test method, when normality is satisfied through the normality test of similarity distance data (when the data have a bell-shaped shape, that is, a normal distribution based on the mean value). A statistical test method is used to verify the difference, and if normality is not satisfied, the non-parametric Mann-Whitney U-test statistical test method is used.

An experiment was performed to verify the evaluation method according to the present disclosure as described above.

The raw data used in the above method includes a face front image for extracting fine movements and empathy questionnaire data for classifying empathy levels. These data can be obtained through experiments, and the experimental method used in this patent is as follows.

Among the 60 subjects who participated in the experiment, a total of 59 subjects were excluded, and there were 27 males and 31 females. The subjects were college students or graduate students who had no health problems and did not wear glasses, who voluntarily participated in the experiment.

The research hypothesis of the experiment is as follows. There will be differences in the similarity of facial fine movements according to the user's content sympathy. Accordingly, the independent variable is the user's empathy for the content, and the dependent variable is the facial fine movement.

As for the experimental procedure, a questionnaire was conducted to evaluate subjective empathy for the video while watching a 4-minute emotional-evoking video and taking a 3-minute break. The above process was repeated for 4 images, and the order of stimulation images was presented at random.

The content used in the experiment consisted of a total of four emotion-inducing videos: an image that induces unpleasant-arousal emotions, an image that induces pleasant-arousal emotions, an image that induces pleasant-relax emotions, and an image that induces discomfort-relaxation. . In order to verify whether the video content that induces a specific emotion in the viewer actually induces the emotion, arousal and positivity were evaluated on a 5-point Likert scale for each of the four emotional stimuli for 227 people, and the average of arousal and positive As a result of calculating the score, it was clearly verified that each emotion was induced.

Although shown and described in relation to specific embodiments in the present disclosure, it is understood in the art that the present invention can be variously improved and changed without departing from the spirit or field of the present invention provided by the following claims. It is intended to be clear that those with ordinary knowledge can easily know.

Claims (14)

displaying video content including a character to a viewer through a display device;
photographing the viewer's face image with a camera, and acquiring the face image of the character using an image capturer;
tracing the facial features of the characters and the viewers from the facial images of the characters and the viewers through an image analyzer, respectively, and extracting motion data of the features;
extracting a heart band signal of 0.75 to 2.5 Hz by frequency analysis from the key point motion data by a fine motion information analyzer, and evaluating the similarity between the key point motion data of the person and the viewer by the Euclidean distance; And
Comparing the degree of similarity to an arbitrarily set reference value using an empathy evaluator to determine whether the viewer sympathizes with the character; delete According to claim 1,
The key point movement data is extracted from the coordinate movement of the key point between the front and rear frames of the facial image, the empathy evaluation method for content. 4. The method of claim 3,
The key point movement data is extracted from the movement in the longitudinal direction (y-axis) among the coordinate movements of the key point, the empathy evaluation method for content. delete delete 5. The method of any one of claims 1, 3 or 4
The similarity is obtained from the periodicity of the characteristic point movement data of the character and the viewer, a method for evaluating empathy for content a display device for displaying image content including characters;
a camera for acquiring a facial image of a viewer viewing the video content;
an image analyzer for tracking feature points of the face from the facial images of the characters and viewers through an image analyzer, and extracting motion data of the feature points;
By evaluating the similarity between the character and the viewer's feature point motion data, a heart band signal of 0.75 to 2.5 Hz by frequency analysis is extracted from the feature point motion data, and the degree of similarity between the character and the viewer's feature point motion data a fine-motion information analyzer that evaluates to by Euclidean distance; And
Empathy determination apparatus comprising a; empathy determiner for determining whether the viewer sympathizes with the video content by detecting the synchronization of the movement data of the viewer's facial features with respect to the character by comparing the degree of similarity to an arbitrarily set reference value. delete 9. The method of claim 8,
In the image analyzer, the movement of the feature extracted from the facial image of the person and the viewer is extracted from the coordinate movement of the feature between the front and rear frames of the facial image. 11. The method of claim 10,
The image analyzer extracts from the movement in the longitudinal direction (y-axis) of the coordinate movement of the feature, empathy determination device. delete delete 12. The method of any one of claims 8, 10 or 11,
The empathy determiner extracts the similarity from the periodicity of facial micro-movement data of the character and the viewer, the empathy determination device.

Images