KR101846350B1 - evaluation method and system for user flow or engagement by using body micro-movement - Google Patents

Abstract

A method and system for determining that a user has a high level of immersion is described. When the variation value (%) of the n FL (x value) and the ln HF (y value) acquired from the user increases linearly with Y = 0.5614 * x It is judged that the degree of immersion is high, and if one of the two is not satisfied with the rule base, it may be determined that the degree of immersion is low.

Description

[0001] The present invention relates to a method and apparatus for evaluating a user's immersion degree using a human body micro-

Described is a method and apparatus for evaluating the immersion degree of a user (subject) using the body motion.

Flow or engagement refers to the state of concentrating all of your minds on a particular object, with all the stimuli, obstacles and mental thoughts around you blocked (Csikszentmihalyi, 1997). The immersion induces a sense of unity with the immersive object, enabling the memory enhancement or the quick acquisition of the object. The user is provided with information through a real environment or various media contents. However, this does not provide an understanding of the process by which users receive and acquire information through the provision of unidirectional information. The extent of the user's immersion in the information provided by the media content or the actual environment is closely related to the efficient delivery of the information provided. When the user is immersed in the content provided by the user, the user can more easily contact and acquire information provided by the content. Therefore, increasing the user's level of involvement of contents is a major factor that can increase the effect of contents.

In order to increase the user's commitment, the development of technology that can quantitatively evaluate the degree of user's commitment should be preceded. If it is possible to quantitatively evaluate the user's immersion level, it is possible to provide feedback for increasing the user's immersion through changes in various internal and external factors such as contents contents, contents presentation methods, and contents presentation environments. This means a two-way interaction between the user and the content. It is possible to present the information of contents more efficiently to the user than the existing unidirectional interaction.

Csikszentmihalyi, M. (1997). Finding flow: The psychology of engagement with everyday life. Basic Books. Ekman, P. (1972). Universal and cultural differences in facial expressions of emotions. In J. K. Cole (Ed.). Nebraska symposium on motivation. Lincoln: University of Nebraska Press, 207-283. Pan, J., and Tomkins, W. J. (1985). A real-time QRS detection algorithm. Biomedical Engineering, IEEE Transactions on, (3), 230-236.

The present invention proposes a method and system for quantitatively evaluating a user's degree of immersion using human motion.

The method of evaluating human immobility based on human body according to the present invention:

Capturing a fine movement of a user exposed to a stimulus to generate image data;

Performing facial tracking on a subject while generating the image data;

Performing image data processing including spatial decomposition on the image data;

Extracting fine motion information of the subject's face through the image data processing;

Extracting heartbeat information using the fine motion information;

Obtaining RRI (R-peak to R-peak intervals) data from the heartbeat information;

Extracting HRV data from the RRI data;

Calculating power values of a high-frequency band (HF) higher than a predetermined low-frequency band (LF) and a low-frequency band (LF), respectively;

And determining whether the user is immersed in the stimulus using the two power values.

According to an embodiment of the present invention, natural logarithms ln LF and ln HF can be calculated from the two power values, and the user's immersion feeling can be evaluated using the value of the change amount.

According to another embodiment of the present invention, a mean can be obtained from RRI data, and the immersion feeling of the user can be evaluated using the variation of the mean.

According to a specific embodiment of the present invention, when the variation (%) of the average of the RRI is compared with the reference value of 1.583%, it can be evaluated that the user's degree of immersion is higher when it is higher.

According to a specific embodiment of the present invention, when the variation value (%) of the ln FL (x value) and the ln HF (y value) is located above a straight line equation of Y = 0.5614 * x, Can be judged to be high.

That is, the present invention extracts HR information from the somatosensory body, uses the RRI mean obtained therefrom, and ln LF and ln HF of the Normaized HRV as significant variables for evaluating the immersion.

According to an embodiment of the present invention, the step of extracting the fine motion information and the step of extracting the heartbeat information include:

A face tracking step;

A spatial decomposition step;

A Neuro Filter step;

A temporal processing step;

A reconstruction step;

A frame difference averaging step;

A smoothing filter step; And

And a sliding peak detection step.

According to a specific embodiment of the present invention, the HRV data may be obtained through resampling and FFT analysis of the RRI.

The system for performing the method of evaluating the human immersion-based user engagement according to the present invention comprises:

A camera for generating the image data;

An image processing unit for processing image data from the camera;

And an analysis unit for analyzing the data from the image processing unit and evaluating the degree of immersion of the user.

Figure 1 illustrates the experimental method of the present invention.
Figure 2 illustrates the experimental procedure of the present invention.
FIG. 3 shows an electrocardiogram measurement point in the immersion degree evaluation method of the present invention.
4 illustrates an electrocardiogram signal processing procedure in the method of assessing immersion in the present invention.
FIG. 5 shows the result of the RRI analysis (* p <.05, ** p <.01, *** p <.001) in the method of evaluating the degree of involvement according to the present invention.
FIG. 6 shows the results of the RRI variation analysis (* p <.05, ** p <.01, *** p <.001) in the method of evaluating the degree of engagement according to the present invention.
FIG. 7 shows the result of ln LF analysis (* p <.05, ** p <.01, *** p <.001) in the method of evaluating the degree of immersion according to the present invention.
Figure 8 shows ln HF analysis results (* p < .05, ** p < .01, *** p < .001) in the method of assessing immersion according to the present invention.
9 shows the results of analysis of the change in ln LF (* p <.05, ** p <.01, *** p <.001) in the method of measuring the degree of involvement according to the present invention.
Figure 10 shows the results of analysis of ln HF variation (* p < .05, ** p < .01, *** p < .001) in the method of evaluation of immersion according to the present invention.
FIG. 11 shows the result of the autonomic nervous system balance analysis (high engagement) of each subject in the immersion degree evaluation method according to the present invention.
12 shows the results of the autonomic nervous system balance analysis (low engagement) of each subject in the immersion degree evaluation method according to the present invention.
13 illustrates an RRI-based immersion rating rule-base in the immersion rate evaluation method according to the present invention.
Figure 14 illustrates the rule-based evaluation of ln LF and ln HF-based immersion in the method of assessing engagement according to the present invention.
15 is a flowchart of an evaluation method based on an immersion evaluation integrated rule base in the immersion evaluation method according to the present invention.
FIG. 16 illustrates RRI-based rule-based accuracy verification results in an immersion rate evaluation method according to the present invention.
17 shows the result of the rule-base accuracy verification based on Normalized HRV in the method of evaluating the degree of engagement according to the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Reference will now be made in detail to the preferred embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the like elements throughout. In the present invention, a user or a subject who has received a specific stimulation by various contents or the like is intended, and the subject is to evaluate the immersion feeling for a specific stimulus.

The system for performing the method of the present invention includes a camera for generating the image data, an image processing unit for processing image data from the camera, and an analyzing unit for analyzing the data from the image processing unit and evaluating the user's degree of immersion Such a system may be implemented by a PC-based system equipped with a moving image photographing apparatus.

First, an experimental method and the like for verifying the human immersion-based user engagement evaluation method according to the present invention will be described first.

1. Experimental Method

1.1 Subjects

Subjects participated in this study were 30 students from Sangmyung University and graduate school (mean age: 26.24 ± 4.2). Subjects participated in the experiment were autonomic nervous system, and those without cardiovascular abnormality or history. Prior to participating in the experiment, caffeine, alcohol, and smoking, which could affect autonomic and cardiovascular fitness, were limited and sufficient sleep was provided to minimize fatigue on the day of the experiment. In addition, the subjects were informed of the outline of the experiment except for the purpose of the study, and then the consent of the volunteer participant was obtained and the motivation to participate actively in the experiment by paying the prescribed amount Respectively.

1.2. Experimental Procedure

Sixty subjects participating in the experiment were randomly divided into 30 groups of two in each group. Two subjects in each group were divided into a leader and a follower. The reader was asked to follow his facial expressions in six facial expressions (Ekman, 1972), which were presented in the system in six Ekman basic emotions (happiness, sadness, surprise, disgust, horror, anger). The follower instructed the reader to look at or follow the face of the leader. The above task was repeated by dividing the face to face condition in which the face is faced and the screen condition in which the other side is viewed through the screen.

In this experiment, we defined LE as a low engagement state in which the reader only looks at the reader 's face under screen conditions and the immersive feeling of the follower' s facial expression in the face - to - face condition (HE, high engagement). In addition, an introduction task introducing facial expressions and a practice task for training facial expressions in advance are included to enable facial expressions to be learned in advance and enable natural facial expressions. Each task included 30 seconds of rest and 5 seconds of rest between facial expressions. Detailed experimental methods and procedures are shown in Figs. 1 and 2, and image processing is shown in Fig.

1.3. Data collection

The present invention extracts the heartbeat information from the somatosensory.

In the method for extracting fine motion data according to the present invention, the processing sequence includes the following 10 steps in total. Figure 3 illustrates step-by-step operational scenarios.

end. In the video input (Audio Input) step S11,

The subject is photographed with a moving image camera to generate continuous image data. At this time, the photographing area is the upper half of the body including the face as shown in Fig.

I. Face tracking or tracking (Face Tracking) step S12

The image information is separated through facial recognition using OpenCV (Open Computer Vision) in order to extract the minute movement of the human body from the head using the image data input by using the image camera.

OpenCV is C. Originally developed, it can be used on multiple platforms, such as Wu. OpenCV is a library focused on real-time image processing.

The OpenCV-based facial recognition system refers to a computer-aided application program that automatically identifies each person through a digital image. This is done by comparing the facial database with the selected facial features that appear in the live image.

In FIG. 3, a rectangle appearing in the original image in step S12 represents the tracking area of the face portion. This enables tracking of the face portion in response to the movement of the user.

All. In the spatial decomposition step S13,

It is a well-known spatial segmentation technique that separates each frequency by Gaussian blur and downsample.

The image shown in S13 of FIG. 3 is a result of the state in which the images are separated in space.

The above formula, α is an amplification factor value of the image, β is a time, and the value of the image filter as a spatial frequency band, x is the horizontal axis (x) and the position of the image, t is a time, δ (t) is Is the mixture ratio value of the input image and the motion image, and I 'is the movement degree of the moving image.

la. In the Neuro Filter step S14,

In the neurofilter phase, a frequency band that can be extracted from an image based on a general bio signal is selected from a frequency band having correlation between biological signals, and a frequency band is selected And acquire image information of the selected band.

For example, when signal processing of general PPG data is performed, data analysis is performed with a frequency component of 0.4 to 1.3 Hz. In order to acquire a PPG-like signal from the image, similar data is analyzed through frequency analysis of the same band or nearby band .

hemp. Temporal Processing Step S15:

After separating the image into the frequency space of the Neuro Filter band, the difference value is extracted using the temporal processing of the frequency band of the main component of the separated space, The values of the corresponding frequency components are separated and extracted.

bar. In the reconstruction step S16,

By separating the components of the separated space using only the frequency components of the Neuro Filter band using time and restoring the separated component values into the existing image through a certain amount of amplification, So that the data value of the component can be generated.

In the above equation, α is the amplification ratio value of the image, δ ( t ) is the mixing ratio value of the input image and the motion image, and λ is the wavelength band value of the image space.

four. The frame difference averaging step (S17)

By calculating the difference value of the average of one frame of the motion data of the image which is measured every time the data value of the separated component is measured (based on 30 fps), the difference value of motion between the previous state and the current state is calculated, The amount is extracted. Here, the average of one frame represents the amount of fine motion of one frame.

Ah. In the smoothing filtering step S18,

When the extracted fine motion is extracted as data, noise is included in the motion and the signal is roughly distorted. Therefore, it is difficult to detect the peak, thereby processing the data to remove the noise and improve the accuracy of the peak detection.

SMA is the moving average, SMAtoday and SMAyesterday are the moving average values of different specific dates, Pm is the fine value of the current frame, and n is the window size of the moving average.

character. Sliding Peak Detection (S19, S20)

It removes noise and receives processed data for peak detection, and keeps the peak data per frame continuously sliding based on the window size of 30 seconds to minimize the influence of motion and data. And extracts a BPM (bit per minute) signal as heartbeat information (PPG).

In the equation above i is 0, 1, ... , n, etc., and w is the window size.

1.4 Signal Processing

The PPG signal obtained in the above procedure detects the R-peak through the QRS detection algorithm (Pan and Tompkins, 1985). R-peak (R-peak to R-peak interval) was extracted from the detected R-peak by using the difference of the normal R-peak interval. To analyze the heart rhythm pattern (HRP), the beat per minute (BPM) can be calculated using 60 / RRI and the SDNN (standard deviation normal to normal) is extracted using the standard deviation of the normal RRI There is also water.

The heart rate information (PPG) data measured through the above procedure detected R-peak through the QRS detection algorithm (Pan & Tomplins, 1985). The detected R-peaks were obtained by calculating the R-peak to R-peak intervals (RRI) by calculating the time interval between R-peaks based on the data of normal heartbeat intervals, except for abnormal heart rate data. RRI data was re-sampled at 2 Hz to convert to time-series data and HRV (Heart Rate Variability) spectrum data was extracted through Fast Fourier Transform (FFT) analysis. The extracted data LF by - - (0.4 Hz High Frequency, 0.15) respectively, were calculated power value of each of the power value of the band has taken the natural logarithm ln LF and ln HF (very Low Frequency, 0.0033 0.04 Hz) band and the HF Respectively. The computed values were plotted in coordinates consisting of nine domains on the ln LF and ln HF axes to confirm the autonomic nervous system balance. The detailed signal processing procedure is as shown in FIG. 4 as described above.

2. Experimental results

In this experiment, the viewing expression task of screen condition is defined as LE (low engagement) and the imitation task of face to face condition is defined as high engagement (HE). The data of all task intervals are compared with the reference interval to calculate the change rate and compared the results of the immaturity (Ratio of change).

2.1. RRI (R-peak to R-peak Intervals)

As a result of RRI analysis, HE showed a significant increase in RRI compared to reference ( t = -3.099, p = .004), LE showed a pattern of decreasing RRI compared to reference but not statistically significant t = 1.237, p =. 226). Analysis of change showed that HE was significantly increased compared to LE ( t = 7.248, p = .000). Detailed analysis results are shown in Fig. 5 and Fig.

2.2. Normalized HRV ( ln LF &lt; ln HF)

Normalized HRV analysis showed that HE had a significant increase in ln LF compared to reference ( t = -7.128, p = .000), LE had a significant decrease in ln LF relative to reference ( T = 4.026, p = .000), respectively. Detailed analysis results are shown in Fig.

In addition, HE showed a significant decrease in ln HF compared to reference ( t = 5.227, p = .000) and LE showed a significant increase in ln HF relative to reference ( t = -5.900 , p = .000). Detailed analysis results are shown in Fig.

In the analysis of change, ln LF showed a statistically significant increase in HE compared to LE ( t = 6.312, p = .000) and ln HF showed a statistically significant decrease in HE compared to LE ( T = -9.092, p = .000). Detailed analysis results are shown in FIG. 9 and FIG.

ln LF and ln HF were plotted in nine domains. As a result, HE decreased in the direction of decreasing LF (sympathetic nervous system) activity and HF (parasympathetic nervous system) activity relative to the reference section And LE was shifted toward LF (sympathetic nervous system) activity and HF (parasympathetic nervous system) activity in comparison with the reference period. Detailed analysis results are shown in FIGS. 11 and 12. FIG.

2.3. Immersion rating rule - base setting

The immersion evaluation rule base was set based on the change analysis result. The result of analyzing the RRI change amount of each subject according to HE and LE conditions is as shown in FIG. For each subject, we set the threshold that has the greatest discretion to distinguish between HE and LE. Based on this, we set the rule-base to judge the case where the change of the RRI from the reference increases greatly from 1.53% to HE, and the opposite case to the rule-base.

The result of the analysis of the change amounts of ln LF and ln HF of each subject (user) plotted according to the HE and LE conditions is as shown in Fig. For each subject, we set the threshold that has the greatest discretion to distinguish between HE and LE. (Y = 0.56146 × X), which is a rule-based equation of the straight line passing through the midpoint between the two closest points of LE and HE data and zero point. Thus, we set the rule base to judge HE as the difference between the reference values of n LF and ln HF and the LE as the opposite case.

A rule-base for evaluating the immersion degree by integrating the two rule-bases is set, and the integrated rule-base is shown in FIG.

3. Accuracy Verification Result

In order to verify the accuracy of the rule-base, 15 subjects participated in the verification experiment. As a result of evaluating the immersion level through RRI change, the rule-based test results showed that all 15 of the 15 cases were judged as HE, and the LE judged 13 cases of the 15 cases as LE and the remaining cases were judged as HE Respectively. Detailed study results are shown in Fig.

In the result of evaluating the immersion degree based on the normalized HRV, the HE judged all 15 cases out of 15 cases, and the LE judged 14 out of 15 cases as LE and the other case judged as HE Respectively. Detailed study results are shown in Fig.

Finally, all of 15 cases of HE were judged to be HE, and the classification accuracy of HE was 100% (15/15 * 100 = 100%). LE was judged to be 12 cases out of 15 cases and LE was judged to be 3 cases. The classification accuracy of LE was 80% (12/15 * 100 = 80%). The evaluation accuracy of the total commitment was 90% (27/30 * 100 = 90%).

4. Conclusion

An object of the present invention is to provide a method of quantitatively evaluating a user's immersion level based on an electrocardiogram signal. Based on the results of the study, it was confirmed that the RNI mean and the normalized (Normaized) HRV ln VFL and ln HF were significant variables for assessing immersion. The following is the case where the user is determined to have a high level of immersion (in the opposite case, the user's immersion is low).

1. When the variation value (%) of the RRI average (mean) acquired from the user greatly increased by 1.583%

2. When the variation value (%) of n VFL (x value) and ln HF (y value) acquired from the user is above the linear equation of Y = 0.5614 * x

3. Judge that the two rules-base satisfies the immersive state, and judges that the immersion is low if one of the two rules does not satisfy the rule-base

The user involvement evaluation method proposed in the present invention can evaluate the immersion degree of a user's content in real time in a media medium or a real environment. It is expected that this will enable interactive interaction that can increase user 's immersion through various feedback based on user' s immersive state, rather than one - way interaction between user and contents. It is expected that the above-described invention will be applied to various industrial fields and enable a more effective information providing effect through contents.

While this invention has been described in connection with what is presently considered to be the most practical and preferred embodiment, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the invention. Therefore, the true scope of protection of the present invention should be defined only by the appended claims.

Claims (11)

Capturing a fine movement of a user exposed to a stimulus to generate image data;
Performing facial tracking on a subject while generating the image data;
Performing image data processing including spatial decomposition on the image data;
Extracting fine motion information of the subject's face through the image data processing;
Extracting heartbeat information using the fine motion information;
Obtaining RRI (R-peak to R-peak intervals) data from the heartbeat information;
Extracting HRV data from the RRI data;
Calculating power values of a low frequency band (LF) in a predetermined range and a high frequency band (HF) in a higher frequency range than the low frequency band (LF) in the HRV data; And
The amount of change in ln HF When the low frequency band (LF) and is calculated from the power value of the high frequency band (HF) as the natural logarithm ln LF and ln HF, substituting the ln LF in the x value of <type> below the linear equation And determining that the degree of engagement of the user is high if the value (%) is greater than Y. The method of claim 1,
<Expression>
Y = 0.5614 * x
And determining whether the user is immersed in the stimulus using the power values of the low frequency band (LF) and the high frequency band (HF). delete The method according to claim 1,
A method for evaluating the user's immersion level based on human body, wherein a mean is obtained from RRI data, and a feeling of immersion of the user is evaluated by using a change amount of the mean. The method of claim 3,
And evaluating that the user's degree of immersion is high when the change amount (%) of the average of the RRI is higher than the reference value of 1.583%. delete The method according to any one of claims 1, 3, and 4,
Extracting the fine motion information and extracting the heartbeat information comprises:
A face tracking step of tracking the face portion while photographing the face with a video camera;
A spatial decomposition step of separating image data obtained from the image camera by frequency space;
A neurofilter for selecting a frequency band related to a bio-signal from components of the frequency space obtained in the spatial separation step;
A temporal processing step of separating and extracting the values of the components of the frequency band over time;
Reconstructing the time-processed component value and reconstructing the reconstructed image to form a data value of the frequency component of the fine motion;
A frame difference averaging step of calculating a difference value between the frames of the image according to the fine movement;
A smoothing filter step of removing noise from data on fine motion; And
And a sliding peak detection step of detecting a peak by a sliding window technique. delete The method according to any one of claims 1, 3, and 4,
Wherein the HRV data is acquired through resampling and FFT analysis of the RRI. 7. A system for performing the method of any one of claims 1, 3, or 4,
A camera for generating the image data;
An image processing unit for processing image data from the camera;
And an analysis unit for analyzing data from the image processing unit and evaluating the user's degree of immersion. delete delete

Images