KR102389361B1 - evaluation method and system for user flow or engagement by using body micro-movement - Google Patents

Abstract

The cases in which it can be determined that the user's immersion level is high are as follows. In the opposite case, it means that the user's immersion level is low. When the change value (%) of the RRI mean obtained from the user increases by 1.583%, the change value (%) of n VLF (x value) and ln HF (y value) obtained from the user is a straight line of Y = 0.5614*x If it is located above the equation of , the case of satisfying the two rule-bases is judged as a high level of immersion, and if at least one of the two rules does not satisfy the rule-base, it is determined as a low level of immersion. .

Description

Method and device for evaluating user's immersion using body micro-movements {evaluation method and system for user flow or engagement by using body micro-movement}

A method and apparatus for evaluating the immersion of a user (subject) using body movements will be described.

Flow or engagement refers to a state of concentrating all of one's mind on a specific object in a state where all surrounding stimuli, obstacles, and mental distractions are blocked (Csikszentmihalyi, 1997). Immersion induces a sense of unity with the object to be immersed in, enabling the improvement of memory or rapid acquisition of the object. Users are provided with information through a real environment or various media contents. However, this is a one-way provision of information, and the understanding of the process of receiving and acquiring information is not taken into consideration. The degree of user's immersion in media content or the information provided in the real environment is closely related to the efficient delivery of the provided information. When the user is immersed in the provided content, it is easier for the user to more efficiently access and learn the information provided by the content. Therefore, increasing the user's immersion in the content is a major factor that can increase the effect of the content.

In order to increase the user's immersion, first, the development of a technology that can quantitatively evaluate the user's level of immersion must be preceded. If it is possible to quantitatively evaluate user immersion, feedback can be presented to increase user immersion through changes in various internal and external factors such as content content, content presentation method, and content presentation environment. This implies a two-way interaction between the user and the content. Content information can be presented to the user more efficiently compared to the existing one-way interaction.

Csikszentmihalyi, M. (1997). Finding flow: The psychology of engagement with everyday life. Basic Books. Ekman, P. (1972). Universal and cultural differences in facial expressions of emotions. In J.K. Cole (Ed). Nebraska symposium on motivation. Lincoln: University of Nebraska Press, 207-283. Pan, J., and tompkins, W. J. (1985). A real-time QRS detection algorithm. Biomedical Engineering, IEEE Transactions on, (3), 230-236.

The present invention proposes a method and apparatus that can quantitatively evaluate a user's immersion level using body movements.

Method for evaluating user immersion based on fine body movements according to the present invention: Silver
generating image data by photographing a user's micro-movements exposed to stimuli;
performing face tracking on the subject while generating the image data;
performing image data processing including spatial decomposition on the image data;
extracting fine motion information of the subject's face through the image data processing;
extracting heart rate information by using the fine motion information;
acquiring RRI (R-peak to R-peak Intervals) data from the heart rate information;
extracting HRV data from the RRI data;
calculating power values of a high frequency band (HF) higher than a predetermined range of a low frequency band (LF) and a low frequency band (LF) from the HRV data, respectively;
and determining whether the user is immersed in the stimulus using the two power values.
According to an exemplary embodiment of the present invention, natural logarithms ln LF and ln HF are calculated from the two power values, and the user's sense of immersion can be evaluated using these values.
According to another embodiment of the present invention, the user's sense of immersion may be evaluated by obtaining a mean from the RRI data and using the change amount of the mean.
According to a specific embodiment of the present invention, when the average change rate (%) of the RRI is higher than the reference value of 1.583%, it can be evaluated that the user's immersion is high.
According to a specific embodiment of the present invention, when the rate of change (%) of the ln LF (x value) and ln HF (y value) is located above the equation of the straight line of Y = 0.5614*x, the user's immersion high can be judged.
That is, in the present invention, heart rate information is extracted from body micro-movements, and the RRI mean and ln LF and ln HF of the normalized HRV obtained therefrom use significant variables to evaluate the immersion.
According to an embodiment of the present invention, the step of extracting the micro-motion information and the step of extracting the heart rate information:
Face Tracking (Face Tracking) step;
Spatial Decomposition;
Neuro Filter (Neuro Filter) step;
Temporal Processing;
Reconstruction (Reconstruction) step;
Frame difference averaging (Frame Difference Average) step;
Smoothing filter (Smoothing Filter) step; And
It may include; a sliding peak detection (Sliding Peak Detection) step.
According to a specific embodiment of the present invention, the HRV data may be obtained through resampling and FFT analysis of the RRI.
A system for performing the user immersion evaluation method based on fine body movements according to the present invention:
a camera that generates the image data;
an image processing unit for processing image data from the camera;
An analysis unit that analyzes the data from the image processing unit to evaluate the degree of immersion of the user: may include.

delete

delete

delete

1 illustrates the experimental method of the present invention.
Figure 2 illustrates the experimental procedure of the present invention.
3 shows an electrocardiogram measurement point in the immersion evaluation method of the present invention.
4 illustrates an electrocardiogram signal processing procedure in the immersion evaluation method according to the present invention.
5 shows 5. RRI analysis results (* p < .05, ** p < .01, *** p < .001) in the immersion evaluation method according to the present invention.
6 shows the RRI variation analysis results (* p < .05, ** p < .01, *** p < .001) in the immersion evaluation method according to the present invention.
7 shows the ln VLF analysis results (* p < .05, ** p < .01, *** p < .001) in the immersion evaluation method according to the present invention.
8 shows the ln HF analysis results (* p < .05, ** p < .01, *** p < .001) in the immersion evaluation method according to the present invention.
9 shows the results of ln VLF change analysis (* p < .05, ** p < .01, *** p < .001) in the immersion evaluation method according to the present invention.
10 shows the results of analysis of ln HF variation (* p < .05, ** p < .01, *** p < .001) in the immersion evaluation method according to the present invention.
11 shows the autonomic nervous system balance analysis result (high engagement) of each subject in the engagement evaluation method according to the present invention.
12 shows the autonomic nervous system balance analysis results (Low engagement) of each subject in the engagement evaluation method according to the present invention.
13 illustrates an RRI-based immersion evaluation rule-base in the immersion evaluation method according to the present invention.
14 illustrates an engagement evaluation rule-base based on ln VLF and ln HF in the engagement evaluation method according to the present invention.
15 is a flowchart of an evaluation method based on an immersion evaluation integrated rule-base in the immersion evaluation method according to the present invention.
16 illustrates an RRI-based rule-based accuracy verification result in the immersion evaluation method according to the present invention.
17 shows a result of verifying the accuracy of a rule-based accuracy based on a Normalized HRV in the immersion evaluation method according to the present invention.

Hereinafter, an embodiment of a method and apparatus for obtaining heart rate information from body micro-movements and evaluating a user's immersion level using the heart rate information obtained therefrom according to the present invention will be described in detail with reference to the accompanying drawings.

1. Experimental method

1.1 Subject

The subjects who participated in this study were 30 students of Sangmyung University and graduate school (15 male and female each, mean age: 26.24±4.2). Subjects participating in the experiment were those who had no autonomic nervous system and cardiovascular abnormalities or a history. Before participating in the experiment, caffeine, alcohol, and smoking, which could affect the autonomic and cardiovascular system, were restricted, and fatigue was minimized on the day of the experiment through sufficient sleep. In addition, after explaining the general details of the experiment except for the research purpose from the subject, consent was obtained from the subject for voluntary participation, and the motivation to actively participate in the experiment by paying a certain amount in return for participating in the experiment was given.

1.2. Experimental procedure

The 60 subjects who participated in the experiment were randomly divided into 30 groups, two in each group. Two subjects in each group were divided into a leader and a follower, respectively, and the experiment was conducted. The leader saw the facial expression in the six Ekman basic emotions (happiness, sadness, surprise, disgust, fear, and anger) presented in the system and had them follow their own facial expressions (Ekman, 1972). Followers were instructed to simply look at or follow the facial expressions made by the leader. The above task was repeated by dividing it into a Face to Face condition, which is performed in a face-to-face condition, and a Screen condition, which proceeds while seeing the other party through the screen. In this experiment, just staring at the leader's expression in the screen condition was defined as low engagement, and in the Face to Face condition, the condition in which followers follow the leader's facial expression was defined as high engagement. was defined as In addition, in order to learn about facial expressions in advance to enable natural facial expressions, an introduction task to introduce facial expressions and a practice task to train facial expressions in advance were included. A 30 second break was included between each task, and a 5 second break was included between facial expressions. Detailed experimental methods and procedures are shown in FIGS. 1 and 2 .

1.3. data collection

The present invention extracts heart rate information from body micro-movements.

In the method of extracting fine movement data according to the present invention, the processing sequence includes a total of 10 steps as follows. 3 illustrates a step-by-step operation scenario.

go. Video input (Audio Input) step (S11)

Continuous image data is generated by photographing a subject with a video camera. In this case, the photographing area is the upper body including the face as shown in FIG. 3 .

me. Face Tracking or Tracking Step (S12)

In order to extract minute movements of the human body from the head using the image data input using the image camera, image information is separated through face recognition using OpenCV.

OpenCV (Open Computer Vision) is C. Originally developed by this, it can be used on multiple platforms such as Woo, etc. This OpenCV is a library focused on real-time image processing.

The OpenCV-based facial recognition system refers to a computer-aided application program that automatically identifies each person through a digital image. This is done by comparing selected facial features appearing in the live image with a facial database.

In FIG. 3 , a rectangle appearing in the original image in step S12 indicates the tracking area of the face. In response to the user's movement, tracking of the facial part is performed.

all. Spatial Decomposition step (S13)

As a well-known spatial separation technique for images, the space for each frequency is separated through Gaussian blur and down-sampling.

The image shown in S13 of FIG. 3 is a result of a spatially separated state.

In the above equation, α is the image amplification ratio value, β is the value of the image filtered with time and spatial frequency bands, x is the horizontal axis (x) position value of the image, t is time, and δ ( t ) is It is a value of the mixing ratio between the input image and the moving image, and I' is the degree of movement of the moving image.

la. Neuro Filter Step (S14)

Neurofilter (Neuro Filter step) selects a frequency band that can be extracted from an image based on a general bio signal in separating spatial frequencies of an image, selects a band with correlation between bio signals, and selects a frequency band. That means acquiring the video information of the selected band.

For example, when processing general PPG data, data analysis is performed with frequency components in the range of 0.4 to 1.3 Hz. In order to obtain a signal similar to PPG from an image, similar data is analyzed through frequency analysis of the same or neighboring bands. extract

mind. Temporal Processing Step (S15)

After dividing the image into the frequency space of the neurofilter band, the frequency band of the main component of the separated space is extracted using temporal processing to extract the difference value while the image is in progress (time flows while), the value of the corresponding frequency component is separated and extracted.

bar. Reconstruction step (S16)

Only the frequency components of the Neuro Filter band are separated using time to separate the components of the separated space, and the separated component values are restored to the existing image through a certain amount of amplification, so that the frequency corresponding to the movement that is invisible in detail. Allows you to create data values of the components.

In the above equation, α is the value of the amplification ratio of the image, δ ( t ) is the mixing ratio of the input image and the motion image, and λ is the value of the wavelength band of the image space.

buy. Frame difference averaging step (S17)

By calculating the difference value of the average of one frame of motion data of the image measured every time (based on 30 fps) for the data values of the separated components, the difference between the previous state and the current state is calculated on the average of the overall fine movement. extract the amount Here, the average of one frame represents the amount of fine motion of one frame.

Ah. Smoothing Filtering Step (S18)

When the extracted micro-movement is extracted as data, it is difficult to detect a peak because noise about the movement is included and the signal is roughly distorted.

In the above equation, SMA is the moving average value, SMAtoday and SMAyesterday are the moving average values of different specific days, Pm is the current frame's non-moving value, and n is the window size of the moving average.

ruler. Sliding Peak Detection Step (S19, S20)

Removes noise and receives processed data for peak detection and continuously slides peak data per frame based on a window with a size of 30 seconds to minimize the influence of motion and data Enables extraction of BPM (bit per minute) signals.

1.4 Signal processing

For the PPG signal obtained in the above process, R-peak was detected through the QRS detection algorithm (Pan and Tompkins, 1985). For the detected R-peak, RRI (R-peak to R-peak interval) was extracted using the difference between normal R-peak intervals excluding noise. For heart rhythm pattern (HRP) analysis, heart rate per minute (BPM, beat per minute) can be calculated through 60/RRI, and SDNN (standard deviation normal to normal) is extracted using the standard deviation of the normal RRI. You may.

For the heart rate information data measured through the above process, R-peak was detected through the QRS detection algorithm (Pan & Tomplins, 1985). For the detected R-peak, RRI (R-peak to R-peak Intervals) data were obtained by calculating the time interval between R-peaks based on the data of the heart rate interval, which is the settlement, except for the data of the abnormal heart rate interval. RRI data were re-sampled at 2 Hz to convert to time series data, and HRV (Heart Rate Variability) spectrum data was extracted through FFT (Fast Furier Transform) analysis. For the extracted data, the power values of the VLF (Very Low Frequency, 0.0033 - 0.04 Hz) and HF (High Frequency, 0.15 - 0.4 Hz) bands were calculated respectively. Calculated. The calculated values were plotted on coordinates composed of 9 domains on the ln VLF and ln HF axes to confirm the autonomic nervous system balance. A detailed signal processing process is shown in FIG. 4 .

2. Experimental results

In this experiment, the viewing expression task under the screen condition was defined as low engagement (LE) and the imitation task under the face to face condition as high engagement (HE). The data of all task sections were calculated as the rate of change compared to the data of the reference section, and the results for immersion were compared.

2.1. RRI (R-peak to R-peak Intervals)

As a result of RRI analysis, HE showed a significantly increased RRI compared to the reference ( t = -3.099, p = .004), and the LE showed a decreased RRI compared to the reference pattern, but it was not statistically significant ( t = 1.237, p = .226). As a result of analysis of the amount of change, it was confirmed that HE increased significantly compared to LE ( t = 7.248, p = .000). Detailed analysis results are shown in FIGS. 5 and 6 .

2.2. Normalized HRV ( ln VLF & ln HF)

As a result of normalized HRV analysis, HE showed a significant increase in ln VLF compared to the reference ( t = -7.128, p = .000), and LE confirmed that the ln VLF compared to the reference significantly decreased ( t = 4.026, p = .000). Detailed analysis results are shown in FIG. 7 .

In addition, in HE, ln HF was significantly decreased compared to the reference ( t = 5.227, p = .000), and in LE, it was confirmed that ln HF was significantly increased compared to the reference ( t = -5.900, p = .000). . A detailed analysis result is shown in FIG. 8 .

In the variation analysis results, ln VLF confirmed a statistically significant increase in HE compared to LE ( t = 6.312, p = .000), and ln HF confirmed a statistically significant decrease in HE compared to LE. ( t = -9.092, p = .000). Detailed analysis results are shown in FIGS. 9 and 10 .

As a result of plotting each subject's data in 9 domains for ln VLF and ln HF, HE moved in the direction of decreasing VLF (sympathetic nervous system) activity and increasing HF (parasympathetic nervous system) activity compared to the reference section, and LE was Compared to the reference section, VLF (sympathetic nervous system) activity increased and HF (parasympathetic nervous system) activity moved in a decreasing direction. Detailed analysis results are shown in FIGS. 11 and 12 .

2.3. Immersion evaluation rule-base setting

The immersion evaluation rule-base was set based on the change amount analysis result. The results of plotting the RRI variation analysis results of each subject according to the HE and LE conditions are shown in FIG. 13 . Among the data of each subject, the point having the greatest discrimination power to distinguish between HE and LE was set as a threshold. Accordingly, the case where the change in RRI compared to the reference increased by more than 1.53% was set as HE, and the opposite case was set as LE as the rule-base.

The results of plotting the change amount analysis results of ln VLF and ln HF of each subject (user) according to the HE and LE conditions are shown in FIG. 14 . Among the data of each subject, the point having the greatest discrimination power to distinguish between HE and LE was set as a threshold. Among the LE and HE data, the equation of the straight line passing through the midpoint between the two most adjacent points and the zero point was set as the rule-base (Y = 0.56146 × X). Accordingly, if the value of change compared to the reference of n VLF and ln HF is above the straight line of Y = 0.56146*X, HE and the opposite case were set as LE.

By integrating the two rule-bases, a rule-base for evaluating immersion was set, and the integrated rule-base is shown in FIG. 15 .

3. Accuracy Verification Results

In order to verify the accuracy of the rule-base, 15 subjects participated in the verification experiment. As a result of the rule-based verification, in the result of evaluating immersion through the amount of change in RRI, HE judged all 15 cases out of 15 cases as HE, LE judged 13 cases out of 15 cases as LE, and the remaining 2 cases were judged as HE. was shown. The detailed study results are shown in FIG. 16 .

In the results of evaluation of immersion based on normalized HRV, all 15 cases out of 15 cases in HE were judged as HE, 14 cases out of 15 cases in LE were judged as LE, and the remaining 1 case was judged as HE. The detailed study results are shown in FIG. 17 .

Finally, all 15 cases of HE were judged to be HE, and the classification accuracy of HE was judged to be 100% (15/15*100 = 100%). As for LE, 12 cases out of 15 cases were judged as LE and the remaining 3 cases were judged as HE, and the classification accuracy of LE was determined to be 80% (12/15*100 = 80%). The evaluation accuracy for the overall immersion was confirmed to be 90% (27/30*100 = 90%).

4. Conclusion

It is an object of the present invention to propose a method for quantitatively evaluating a user's immersion level based on an electrocardiogram signal. According to the study results, it was confirmed that RRI mean and ln VLF and ln HF of Normaized HRV were significant variables for evaluating commitment. The cases in which it can be determined that the user's immersion level is high are as follows (in the opposite case, it means that the user's immersion level is low).

1. When the value of change (%) of the RRI mean obtained from the user is greatly increased by 1.583%

2. When the variation value (%) of n VLF (x value) and ln HF (y value) obtained from the user is higher than the equation of the straight line of Y = 0.5614*x

3. A case in which the above two rule-bases are satisfied is judged as a state of high immersion, and a state of low immersion is judged when at least one of the two rule-bases is not satisfied.

The user immersion evaluation method proposed in the present invention can evaluate the user's immersion in content in a media medium or real environment in real time. This is not a one-way interaction between the user and the content, but the content is expected to enable a two-way interaction that can increase the user's immersion through various feedbacks according to the user's immersion state. In addition, the above invention is expected to be applied to various industrial fields to enable a more efficient information provision effect through contents.

Claims (11)

generating image data by photographing micro-movements of a user exposed to stimuli with a camera;
performing face tracking on the subject on the image data by an image processing unit;
Separating the space for each frequency according to Equation 1 below for the image data obtained by facial tracking, extracting the image data in a frequency band in the range of 0.4 to 1.3 Hz;
amplifying the image data of the frequency band and reconstructing an image of a space divided into the frequency band using the image data of the frequency band;
extracting micro-motion information of the subject's face by calculating a difference value between the averages of the front and rear frames of the image through the processing of the image in the separated space by the analysis unit;
extracting heart rate information (PPG) through peak detection of a sliding window with respect to the fine motion information and obtaining RRI (R-peak to R-peak Intervals) data from the heart rate information;
calculating an RRI change amount from the RRI data and calculating a rate of change compared to the change amount of the reference data by Equation 2 below;
extracting HRV data from the RRI data and calculating power values of a high frequency band (HF) higher than that of a low frequency band (LF) and a low frequency band (LF) from the HRV data; And
An evaluation step of judging whether the user is immersed in the stimulus using the two power values, and evaluating the user's engagement as high (High engagement, HE) when the change rate is higher than 1.583% , A method for evaluating user engagement based on body movement.
<Formula 1>

In the above formula,
α: the value of the amplification ratio of the image
β: the value of the image filtered by the temporal and spatial frequency bands
x : the horizontal axis (x) position value of the image
t : time
δ ( t ): the value of the mixing ratio between the input image and the motion image, and
I': the degree of movement of the moved image
<Formula 2>

According to claim 1,
Calculate the natural log ln LF and ln HF from the two power values, and obtain the amount of change (%) of these values, respectively,
When the amount of change (%) of the ln LF (x value) and ln HF (y value) is located above the equation of the straight line of Y = 0.5614 * x, it is determined that the user's immersion level is high. A method for evaluating user engagement based on fine movements. 3. The method of claim 2,
Obtaining the mean (mean) from the RRI data and obtaining the change rate (%) of the mean,
The user's immersion evaluation method based on fine body movement, characterized in that when the average change rate (%) of the RRI is compared with a reference value of 1.583% and higher than this, the user's immersion is evaluated as high. delete delete delete delete According to claim 1,
The HRV data is a body micro-movement-based user immersion evaluation method, characterized in that it is obtained through the resampling of the RRI and FFT analysis. 9. A system for carrying out the method according to any one of claims 1 to 3 or 8, comprising:
a camera that generates the image data;
an image processing unit for processing image data from the camera; And
and an analysis unit that analyzes the data from the image processing unit to evaluate the user's immersion level.
delete delete

Images