WO2017007179A1

WO2017007179A1 - Method for expressing social presence of virtual avatar by using facial temperature change according to heartbeats, and system employing same

Info

Publication number: WO2017007179A1
Application number: PCT/KR2016/007097
Authority: WO
Inventors: 황민철; 원명주; 박상인
Original assignee: 상명대학교서울산학협력단
Priority date: 2015-07-03
Filing date: 2016-07-01
Publication date: 2017-01-12

Abstract

Disclosed is a method for realistically expressing a human avatar by using a facial temperature change according to heartbeats. The method comprises the steps of: detecting ECG data of an actual user in real time; detecting, from the ECG data, a facial temperature change according to heartbeats; and changing the face of the user's avatar in response to the facial temperature change.

Description

A Method of Expressing Social Reality of Virtual-Avatar Using Face Temperature Variation by Heart Rate and Its System

The present invention proposes a method for producing a human avatar or a virtual avatar with enhanced social presence using a change in face temperature according to the heartbeat.

Avatar modeling techniques developed to date have been focused only on the realistic representation of external appearance from the actual anatomical point of view, and the development of a method and system for expressing reality with deeper internal expression is a new task.

The infrared thermal camera detects energy in the form of infrared wavelengths emitted from the surface of the subject and monitors them by expressing them in different colors according to the intensity of radiant heat. The surface temperature of the face varies according to the components of the face such as eyes and mouth, and the face temperature may vary according to the thickness of the skin. In addition, since blood vessels have a higher temperature than other skin tissues, facial surface temperature may vary depending on the distribution of blood vessels. For this reason, research on the relationship between facial temperature and emotional state is underway. Accordingly, the present inventors confirmed the correlation between RRI and facial temperature change by using electrocardiogram (ECG, Electrocardiogram) of autonomic nervous system measurement method, and can infer the face temperature change according to the activity of autonomic nervous system A regression model. Based on the derived regression model, face temperature change according to the emotional state of the user was inferred and applied to the virtual-avatar system.

The present invention proposes a method and a system using the same, which can produce a virtual-avatar in which social reality is enhanced using a change in face temperature according to a heartbeat.

According to the present invention, the realistic representation of the virtual-avatar using the change of the face temperature according to the heartbeat:

Photographing a real user's face in real time;

Detecting a change in face temperature according to a heartbeat change in the face image; And

And changing a face of a virtual avatar correlated with the user in response to the face temperature change.

According to an embodiment of the present invention, the heart rate change may be extracted from the physiological information obtained from the user, and the face temperature change of the virtual-avatar correlated with the user may be synchronized according to the heart rate change.

According to an embodiment of the present disclosure, the heart rate change may be detected from ECG data obtained from the user.

Another embodiment of the present invention may infer a face temperature change of the user from the ECG data through regression analysis, and synchronize the face temperature change of the virtual-avatar with the extracted face temperature change.

According to a specific embodiment of the present invention, the RRI (R-peak to R-peak Intervals) can be detected from the ECG data and the face temperature change of the user can be detected through regression analysis of the RRI.

According to a specific embodiment of the present invention, the user's face temperature change can be expressed by the following equation.

A virtual-avatar realistic representation system according to the present invention comprises: a detection system for extracting physiological information from the user; And an analysis system for detecting a change in the face temperature of the user from the physiological information. And a display system for representing a virtual-avatar having a face model that changes in correspondence with a user's face temperature change from the analysis system by representing the virtual-avatar correlating to the user.

The method proposed in the present invention can be utilized as a new expression element applicable to the 3D model engineer character design. In particular, factors such as facial expressions or gazes expressed by human-avatars or virtual avatars in a virtual environment directly affect the user's sense of reality. In general, however, they are modeled around visual forms or muscle movements. Therefore, through the present invention, in addition to the external elements of the human-avatar face itself, the face temperature change factor due to the internal reaction of the user is an important factor for effectively expressing the human-avatar, and is utilized as a basic study for designing an avatar in a virtual environment. Very large.

1 illustrates a stimulus in accordance with the present invention.

2 illustrates a method of presenting a stimulus in accordance with the present invention.

3 shows the nine ROI regions of the face temperature measured.

4A, 4B, and 4C show the analysis of the correlation between the face temperature response and the ECG response by region.

Figure 5 shows the cardiac response based facial temperature inference model verification results (forehead).

Figure 6 shows the cardiac response based facial temperature inference model verification results (eye left).

Figure 7 shows the cardiac response based facial temperature inference model verification results (no-nose).

Figure 8 shows the cardiac response based face temperature inference model verification results (face to face).

9 shows the virtual sensory evaluation results of the cardiac response based facial temperature sensory expression elements.

Hereinafter, with reference to the accompanying drawings, a description will be given of an embodiment of a realistic expression method of a virtual avatar using a face temperature change according to the heartbeat according to the present invention, and a system for applying the same.

The present invention confirmed the correlation between RRI and facial temperature change by using electrocardiogram (ECG, Electrocardiogram) of the autonomic nervous system, and regression model that can infer the face temperature control response according to the autonomic nervous system activity Derived. Virtual-avatar system by inferring the temperature change response of at least one of the forehead, the eye, the nose, the face, or the entire face according to the emotional state of the user based on the derived regression model Applied to. Here, the virtual avatar may be represented as a human (human), and according to another embodiment, an animal, a virtual creature, for example, an alien or anthropomorphic object may be represented as an avatar.

That is, the present invention by applying the expression element of the facial temperature change according to the actual heart rate change of the user in real time to the virtual-avatar implemented in various forms, to reflect the emotional state of the user and improve the realistic interaction and immersion Here's how to do it.

1. Subject

Participants in this study were 26 university students and 26 general students (13 males, average age: 23.03 years ± 3.27) who had no visual impairment and had uncorrected visual acuity of 0.8 or higher. After fully acknowledging the experimental procedure, they were voluntarily agreed to participate. Subjects were encouraged to take adequate rest prior to the experiment to minimize the effect of sway size reduction due to parasympathetic nerve activation due to drowsiness and fatigue. In addition, it was asked to refrain from exercise, smoking, and caffeine that could affect the sympathetic and parasympathetic nerves.

2. Experimental environment and method

Emotion-induced stimuli used in this study consisted of arousal, relaxation and neutral as audiovisual images. The suitability of the experimental stimulus selection was verified by the fitness test (Relaxation: χ2 [2, N = 26] = 30.769, p = .000, Arousal: χ2 [2, N = 26] = 35.615, p = .000).

The experimental space was 3m wide by 4m long, and the outside noise and noise was less than 30 dB, and the temperature was maintained at 24 ± 1 ℃. All subjects were asked to maintain their physical stability through a 10-minute acclimation time in a sitting position. After that, the gaze at the LED monitor screen of 70 cm distance in a comfortable position. The presented stimulus was randomly presented to the subject to predict it, as shown in Figure 2, to eliminate the order effect.

3. Analysis method

Facial temperature was measured with an infrared thermography camera CX-320U (COX Co. Ltd). Image resolution is 320 x 240 and sampled at 60 frames / second. In the measured image, nine regions (ROIs) were designated to obtain temperature data. As shown in FIG. 3, the nine areas of the face correspond to the forehead, the forehead, the left eye, the right eye, the nose, the mouth, the left cheek, the right cheek, and the entire face. At this time, the temperature data recorded the highest temperature, the lowest temperature, the average temperature and the deviation in the frame unit of the current image. Based on the data, the difference between the temperature information of the current image frame and the next image frame temperature was calculated to calculate the difference in facial temperature with Frame by Frame of the image (thermal to thermal difference).

Electrocardiogram (ECG, Electrocardiogram) was obtained using the Lead-I method. The signal was amplified by the MP 100 & ECG 100C amplifier (Biopac System Inc., USA) and digitized using the NI-DAQ Pad 9205 (National Instrument, USA). The signal is sampled at 500 Hz / second. The measured ECG signal was detected R-peak of the ECG signal using the QRS Detection Algorithm. The detected R-peak was calculated from the RRI (R-peak to R-peak Intervals) by the difference from the adjacent R-peak.

4. Experimental Results

RRI and average face temperature change were used to analyze the correlation between facial and ECG responses. As a result of the correlation analysis, forehead, eye left, nose, and face regions with high correlation were selected as the final effective factors.

A simple linear regression analysis was used to develop and validate a model that predicts facial temperature changes through RRI parameters of ECG responses. In the distribution of independent variable (RRI) and dependent temperature (Facial temperature), the optimal equation was obtained through the least-squares method that finds the least linear line from the values of the actual distributed dependent variable. The slope B is the regression coefficient and A is the constant. R-squared (the coefficient of determination: R2) is the ratio of the variation explained by the regression model to the total variation of the dependent variable. The statistical significance test of the regression model confirmed that the regression model obtained with p = 0.000 was significant.

Equation 1

Equation 2

The facial temperature control response inference model equation using the ECG response estimated by the result is shown in Equation (2), and the detailed pattern is shown in FIGS. 4A, 4B, and 4C.

The reasoning result was mapped to HIS (Hue, Intensity, Saturation) of the virtual avatar and applied the intecity according to one embodiment of the present invention.

To verify the inference model developed by the simple regression analysis method, 10 subjects were tested based on the regression equation derived. As a result of the verification, it was confirmed that the difference between the actual measured face temperature and the predicted face temperature was about 0.00001.

5. Reality Evaluation

The subjects participated in a comfortable position, watching three types of human-avatars (A avatar, B avatar, and C avatar) for 25 seconds each, and at a distance of 60 cm from a 27-inch LED monitor (LG, resolution 1920 x 1080). Presented.

Here, A-avatar is a human-avatar stimulus to which the sensory expression factor is not applied at all, B-avatar is a human-avatar stimulus to which a face temperature is arbitrarily applied to a face region, and C-avatar is The human-avatar stimulus to which the sensory expression element of the heart reaction-based face temperature change confirmed in the experiment of the present invention is applied.

Based on the virtual sensory subjective evaluation model developed in the previous study, a questionnaire about the reality felt by the subject was reported on a 5-point scale after the stimulus was presented. Subjective assessment items consisted of three factors: Visual Presence (VP; 3 item), Visual Immersion (VIm; 7 item), and Visual Interaction (VIn, 4 item). The reliability of subjective evaluation data was secured, including 4 items).

Here, visual presence refers to the degree to which the virtual environment given to the user is perceived, and visual immersion refers to how extremely realistic the virtual environment given to the user is expressed. And visual interaction refers to the degree to which the user can interact with the form or content of each environment through the virtual environment.

Table 1

As a result of evaluating the virtual sensitization of the face response sensory expression factor based on the cardiac response, we confirmed that the presence of the face temperature change inferred through the cardiac response based on the regression model derived from this study was high in human-avatar (p) <0.001)

Table 2 and Figure 9 below shows the virtual sensory evaluation of the heart response based facial temperature sensory expression.

TABLE 2

In the above, exemplary embodiments have been described and illustrated in the accompanying drawings to help understand the present invention. However, it should be understood that these embodiments are merely illustrative of the invention and do not limit it. It is also to be understood that the invention is not limited to the details shown and described. This is because various other modifications may occur to those skilled in the art.

Claims

And synchronizing the facial temperature change of the virtual-avatar correlated with the user according to the heartbeat change acquired from the physiological information obtained from the user.
The method of claim 1,

The heart rate change is detected from the ECG data obtained from the user.
The method of claim 2,

Inferring the change of the user's face temperature from the ECG data through regression analysis, and synchronizing the face temperature change of the virtual-avatar with the inferred change of the user's face temperature. Way.
The method of claim 3,

R-peak to R-peak Intervals (ECR) is detected from the ECG data, and the facial temperature change of the user is detected through regression analysis of RRI.
The method of claim 4, wherein

In the face of the virtual avatar, at least one of the temperature of the forehead, the eye, the nose, and the face is expressed by the following expression. .
In the system for expressing the social reality of the virtual-avatar performing the method of any one of claims 1 to 5,

A detection system for extracting physiological information from the user;

An analysis system for detecting a change in temperature of the face of the user from the physiological information;

And a display system for displaying a virtual-avatar having a face model that changes in response to a change in user face temperature from the analysis system by representing a virtual-avatar correlating to the user. .
The method of claim 6,

The detection system detects ECG data.