KR20170054837A - Method for extracting heart information of facial micro-movement and system adopting the method - Google Patents

Abstract

Descriptions are made with respect to a method and a system for extracting cardiac information by using a facial tremor. The method comprises the steps of: obtaining a facial image from a test subject; mapping multiple peaks for each facial part on the facial image of the test subject by masking (matching) a standard model for defining a standard peak for each facial part with respect to the facial image; extracting a coordinate change value between peaks in an identical facial part from the facial image; continuously extracting facial micro-movement (MM) information including a change value of a centroid of each of action units (AUs) in which the peaks are defined with respect to a human face; selecting MM information of one or more having valid MM information from the AUs; and detecting R-peak to R-peak intervals (RRI) from the selected MM information, and extracting cardiac information of the test subject by using the detected RRI.

Description

Technical Field [0001] The present invention relates to a method and system for extracting cardiac information based on facial fine motion,

The present invention relates to a cardiac information extraction method using a facial fine movement and a system for applying the same, and more particularly, to a cardiac information extraction method and system based on an AU (Action Unit).

As IT technology evolves, many devices around are evolving. As the development of technology and the improvement of the quality of life have been made, the society has been changed into a human - centered society. In the past, technological advances have been made in mechanical aspects such as products and devices. Recently, these technologies are changing to human-centered technologies, especially smartphone technology. As smartphone technology evolved, the peripherals became faster. Among the various peripherals, the development of wearable devices is accelerating. Many devices for measuring and managing human health information have been developed in such wearable devices, and among them, techniques for monitoring the heart rate information and monitoring the exercise amount and health information have been developed. The development of small devices has led to the development of research and products that utilize biometric information in many different fields. Using these technologies, the automotive industry is putting great effort into developing future cars such as smart cars and health cars. However, it is very difficult to obtain accurate data because these devices are operated by touching the sensor to the human body and are affected by movement or motion. Therefore, non-contact sensing technology based on biomechanical mechanisms is required to reduce the burden on movement and wearing comfort.

The minute movements of the human body are invisible to the human eye. Body movements lead to instinctive, unconscious, and physiological changes in cardiovascular and respiratory systems due to gravity. The changes in body position are transmitted to the central nervous system through visual, somatosensory, autonomic, vestibular, and afferent neurons The body moves through the proper response to blood vessels, heartbeats, and respiratory muscles [3]. In particular, the vestibular system is an anatomical organ that plays a role in establishing a sense of balance connected with various organs of the human body. The vestibular system is controlled by various anatomical organs such as the vestibular system, the vestibular system, the vestibular system, and the autonomic nervous system, which are affected by various organs such as autonomic nervous system, cardiovascular system and respiratory system [3].

Fine movements also appear when you create facial expressions. This is referred to as micro-expression, which creates a momentary and unintentional facial expression, or instantly creates facial expressions according to unconscious emotions or reaction states [4]. In other words, it is a motion that is operated by a biomechanical reaction or mechanism, not a facial expression by intention [5]. It is used to realize the feedback system by recognizing emotions and emotional states of people by using such minute movements or tremors [6]. There are a number of studies applying these principles in previous studies. First, there is vibraimage technology that detects information about emotion and emotion. In this technique, a variety of emotion variables are detected by measuring and recognizing the fine movement of the head using a camera [7, 8]. In addition, there is a study on extracting cardiac information using Video Motion technology based on the fine movement of the head [2]. In addition, there are various studies such as heart rate information extraction using RGB color change of video [1].

These studies have limitations that are difficult to apply to everyday life. In order to measure biomechanical information such as heart rate information by measuring minute movements, it is necessary to limit any movement or movement while sitting or standing, but accurate information can be detected. In order to detect information about the actual heartbeat information and the biological response using the minute motion, it is necessary to recognize the person first of all. In addition, there is a need for parts that increase accuracy through detection and data collection at various points rather than information extracted at one point.

US6834115 B2 US6904347 B1 US7352881 B2 EP0735509 B1 WO2002009038 A2 WO2003017186 A1

Poh, Ming-Zher, Daniel J. McDuff, and Rosalind W. Picard. "Advancements in noncontact, multiparameter physiological measurements using a webcam." Biomedical Engineering, IEEE Transactions on 58.1 (2011): 7-11. // MIT Media Balakrishnan, Ganesh, Frederic Durand, and John Guttag. "Detecting pulse from head motions in video." Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on. IEEE, 2013. // MIT CASIL Lee Tae Kyung. Jae Hee. (2006). Effect of Electrostatic System on Autonomic Control. Volume 5. No. 2. Korea Institute of Balance. pp. 329-335. Ekman, P. (2013.12.01). "http://www.paulekman.com/me-historymore/", The website of Paul Ekman Group. Baron-Cohen, S. (1996). Reading the mind in the face: A cross-cultural and developmental study. Visual Cognition, 3 (1), 39-60. Ekman, Paul, and Erika L. Rosenberg. What the face reveals: Basic and applied studies of spontaneous expression using the Facial Action Coding System (FACS). Oxford University Press, 1997. Minkin, V. A., and N. N. Nikolaenko. "Application of vibraimage technology and system for analysis of motor activity and functional state of the human body." Biomedical Engineering 42.4 (2008): 196-200. Minkin, V. A., Georgi P. Gladyshev and Libb Thims (2007). Head movements vibraimage visualization and energetic model of emotions.

The present invention provides a method for extracting cardiac information using a facial fine motion and an apparatus for applying the method.

Method for extracting cardiac information according to the present invention:

Mapping a plurality of vertices to a facial image of the subject by masking (matching) a standard model defining a reference vertex for each facial region on the facial image while obtaining a facial image from the subject;

Extracting a coordinate change value between vertices at the same facial region in the facial image;

Continuously extracting facial micro-movement (MM) information including a change value of a centroid of each of a plurality of Action Units (AU) defined by the vertices of the human face;

Selecting fine motion information of an AU having some effective fine motion (MM) information among the plurality of AUs; And

Detecting RRI (R-Peak to R-Peak Intervals) from the selected fine motion information, and extracting cardiac information of the subject using the detected RRI.

)) 값 중 적어도 어느 하나를 더 포함할 수 있다. According to a specific embodiment of the present invention, the AU facial motion information includes at least one of a facial motion size, an area size, a facial motion angle of a polar concept,

) ≪ / RTI > value).

According to the present invention, the vertex includes X, Y coordinates in the X-Y plane in the direction parallel to the average plane of the subject's face, and Z coordinates perpendicular to the X-Y plane.

According to a specific embodiment of the present invention, the facial motion angle r * Sin (?) Of the polar coordinate concept can be obtained from the change of the Z coordinate perpendicular to the X-Y plane.

According to an embodiment of the present invention, the AU includes at least three vertices.

According to the embodiment of the present invention, the AU facial motion information can be extracted from the motion (change) in at least one of the X-Y plane, the X-Z plane and the Y-Z plane in the X, Y and Z coordinates.

According to a specific embodiment of the present invention, there is provided a method comprising: extracting initial reference data from facial motion information for a predetermined time to obtain facial motion information; Extracting the initial reference data, extracting the facial motion information at regular time intervals, and extracting the RRI from the facial motion information.

According to a specific embodiment of the present invention, the step of selecting fine motion information of an AU having some valid fine motion (MM) information of a plurality of AUs comprises: And selecting a fine motion of

Filtering the selected fine motion information in a predetermined frequency range, and detecting a peak from the filtered fine motion information;

Obtaining the RRI average using the peak; And

And detecting a Beat Per Minute (BPM) signal using the RRI.

A system according to the invention for carrying out the above method comprises:

A moving picture camera for acquiring a facial image from the subject;

An image processing unit for processing an image from the video camera;

And a system for extracting the AU facial motion information using the image data from the image processing unit.

The present invention proposes a method for extracting minute movement of a face as quantified data for each AU and a method for extracting cardiac information using the same. The cardiac information extracted by the present invention shows a pattern very close to the cardiac information directly extracted from the heart as the pseudo-data. It has been confirmed that the present invention can measure cardiac information with excellent results using the non-contact sensing technology, which is a non-contact sensing technique. Especially, although there are many previous studies and similar technologies, it is very difficult to utilize non-contact sensing technology in real life. It is also limited in that it can be measured from the image according to the situation. However, we can confirm that cardiac information can be extracted by using various points of the facial muscles. However, in the present invention, it is necessary to study the lightening of the interlocking module and the more natural tracking technique by applying a tracking module or the like. Therefore, it is expected that cardiac information deduced from the new sensing technique and various facial movement can be applied to real life and related research fields.

1 is a schematic block diagram of an AU-based motion information extraction system according to the present invention.
2 is a flowchart of an AU-based facial motion information extraction method according to the present invention.
Figure 3 shows an example of a standard model applied to the method of the present invention.
FIG. 4 illustrates masking (mapping) the standard model to a facial image acquired from a subject.
Figure 5 shows the result of vertex mapping to the subject image using the standard model.
FIG. 6 illustrates vertex information around the eye, with the left side representing the left eye and the right side representing the right eye.
Figures 7 and 8 illustrate the movement in the AU6 where the ball (cadaver) is raised, the lower eyelid is pressed and the wrinkles of the eye are produced in the method according to the present invention.
Figure 9 illustrates the flow of a method for extracting cardiac information from fine motion in accordance with the present invention.
10 illustrates a flow of a method for extracting respiration information from a fine motion according to the present invention.
11 shows the SD (standard deviation) analysis result of the motion information according to the present invention.
FIG. 12 shows a result of a regression analysis according to motion information according to the present invention.
Figure 13 shows the PPI as a result of data analysis of 24 subjects according to the present invention.
Figure 14 shows BPM as a result of data analysis of 24 subjects according to the present invention.
Figure 15 shows a respiratory sinus arrhythmia (RSA) obtained according to the present invention.
16 shows a case of the number of peaks obtained according to the present invention.

Hereinafter, an embodiment of an AU-based facial motion information extraction method according to the present invention will be described with reference to the accompanying drawings.

The facial motion information extracting method according to the present invention includes a computer system 1 including an input / output device such as a keyboard 14, a mouse 15, a monitor 12, and a main body 11 to which the input / ), And may be implemented by a dedicated system in accordance with an embodiment of the present invention. Such a method of the present invention is not limited in its technical scope by a specific hardware system.

An apparatus or system for performing a method of extracting facial motion information according to the present invention may include a moving picture camera 13, for example, a so-called web cam or a web camera, for photographing the face of the subject 20. In addition, such a system processes an image obtained from a subject through an analysis algorithm provided in software form, extracts facial motion information for each AU using the extracted image, and outputs the result to a database in the computer system 1 and a monitor 12 Lt; / RTI >

2 is a flowchart of an AU-based facial motion information extraction method according to the present invention.

Using vertex tracking technology, we can track vertices of feature points such as eyes, nose, mouth, eyebrows, balls, nose, etc., which have features in the face, based on the 3D model. Respectively.

In the facial tracking and data acquisition process, AU (Action Units) is mapped on the tracked vertex according to the MPEG4 standard, and the face region of each local region is defined to determine the area of the center of gravity or center of gravity, Respectively. The motion information of the mapped centroid region is extracted through the difference value between frames, and the processing is as follows.

First step S21: photographing the face of the subject in real time and inputting the moving image data frame by frame. At this stage, for example, a web camera can be used as the hardware, and the OpenCV library can be used as the software. In this step, a general preprocessing process such as amplifying the input image continuously and removing the noise included therein may be included.

Second Step (S22): Mask the standard model preset in the inputted face image. This process is performed once at the first time, and masking is performed again if necessary when a facial image is missed during future facial tracking.

In the standard model, a plurality of vertices corresponding to various specific parts of the face on the basis of a reference point of the human body surface are appropriately arranged corresponding to the face standard shape or shape. The reference point is, for example, a central part of the hair, and a vertex for specific parts of the face is determined based on the reference point. In the mapping process, the vertices are mapped to the face region of the subject by the portion of the face image based on the standard model. At this time, the comparison region of the subject's face region with respect to the standard model is the face contour, eye, nose, mouth, and the like.

Step 3 (S23): The coordinate change value of the vertices is extracted from the mapped facial image. In extracting the change value at this time, the reference values are compared with the peak values obtained through the masking in the second step.

During this process, face tracking technology available from Visage Technology is available. Visage Technology's facial tracking technology is Candiate-3's 3D model-based tracking technology, which is now used as a standard for various aspects of facial features. It provides information about 137 facial points and provides facial information mapped to the MPEG-4 standard. Therefore, standardized information can be measured.

In addition, 3D model-based tracking technology can reduce the influence of various movements, but it can minimize influence according to facial expression by selecting AU (Action Unit) by using vertex

Step 4 (S24): Map or classify (define) the mapped vertices according to AUs (Action Units).

Step S25: The quantified AU information for each AU including the center point or the centroid of each AU is extracted using the vertices grouped in units of AUs and stored as current AU information, At this time, if there is the AU information stored before that, it becomes the preceding AU information.

Step S6: In this step, it is determined whether or not the preceding AU information exists, and if there is no preceding AU information, the process returns to the first step S21 described above. On the other hand, if there is the preceding AU information, the process proceeds to the seventh step S27.

Step 7 (S27): In this step, micro-movement (MM) of the face is detected. Specifically, the current AU information is compared with the preceding AU information to obtain a change amount of a specific component in the current AU information.

) 등이 있으며, 이들 중 어느 하나가 선택적으로 추출된다. In this step, the specific component to be detected is the center point information per AU, the magnitude of the center point change, the change amount of the area formed by the vertices per AU, the angle of polar motion, and the curve motion r * SIN (

), And any one of them is selectively extracted.

The amount of change in the specific AU information extracted in this manner, that is, the micro-movement, is variously expressed in the deformation of the face of the subject and can be applied to various application fields. For example, a system for extracting or estimating cardiac information of a subject .

Hereinafter, the steps described above will be described in more detail.

3 shows an example of a standard model. In the standard model, the face of the human face with average shape and size is modeled as a network. Each node of the network structure, ie, a vertex, is a part sensitive to facial muscle movement. AUs that define change can be grouped separately.

In FIG. 3, a vertex 136 of the forehead is a reference point for extracting facial expression change amount, and vertices related to eyes, eyebrows, nose, mouth, tail, cheekbones, and jaw line are linked.

FIG. 4 illustrates masking (mapping) the standard model to a facial image acquired from a subject, and FIG. 5 shows a result of vertex mapping to a subject image using a standard model. When mapping is performed as shown in FIG. 5, information of all vertices is extracted based on the vertex of the point-to-point 136. Each vertex information includes the values of X, Y, and Z.

FIG. 6 illustrates vertex information around the eye, with the left side representing the left eye and the right side representing the right eye.

Table 1 below shows the result of mapping AU and vertex, which defines the movement of the facial part, which brings about changes in facial expression.

From the obtained facial image, various kinds of information change amount can be extracted from each AU information, especially AU change.

The midpoints (Cx, Cy) and area (Axy) of the AUs in the XY plane from the facial image can be obtained from the following equations 1, 2, and 3 using the coordinate values of the vertices of each AU. In the equation below, N is the total number of vertices per AU, and x and y are the X and Y coordinates respectively, where i is the number of the vertex in the corresponding AU.

(Cy, Cz), (Cx, Cz), and area (Cz) of the AU in the YZ plane and the XZ plane can be obtained by converting the coordinate values and areas in the XY plane into values in the YZ plane or the XZ plane, Ayz, Axz) can be obtained.

The amount of change in AU information required from the current AU information and the preceding AU information can be obtained using the continuous input facial image in the same manner as described above.

As described above, the amount of AU information change includes the magnitude of the facial movement (|| V ||), the angle of facial motion, the amount of change in the center point, and the amount of change in the area of a specific AU. It can be judged.

In each AU, the magnitude || V || of the nth AU (AUn) is obtained from the following equation.

The facial motion size of AU _n is extracted using the following formula (n = AU number).

(X1, y1) of the centroid of AU _{n in} the previous facial image frame (preceding AU information) and (x2, y2) the center coordinates of AU _n in the current facial image frame (current AU information) , The coordinate change amount V _{n of the center point} is expressed by the following Equation 4.

Therefore, the magnitude (|| V _n ||) of the facial motion vector is calculated as shown in Equation 5 below.

)는 아래와 같이 계산된다. In addition, the center of gravity (radian,

) Is calculated as follows.

는 선행 AU의 벡터

와 현재의 AU의 벡터

내적을 의미한다. 여기에서 두 벡터의 외적을 통해서 회전 방향을 계산할 수 있다. In the above equation

Is the vector of the preceding AU

And the vector of the current AU

It means inner product. Here we can calculate the direction of rotation through the outer product of two vectors.

이면 회전 방향은

는

의 반시계 방향이며,

이면, 회전 방향은

는

의 시계 방향이 된다. In other words,

The direction of rotation

The

Counterclockwise,

, The direction of rotation

The

Clockwise direction.

는 시계방향 또는 반시계 방향으로 움직일 수 있다. Figure 7 illustrates the movement in the AU6 where the ball (cadaver) is raised, the lower eyelid is pressed, and the wrinkles of the eyes are produced. In Fig. 7, a solid line indicates a vector in a frontal facial image, and a dotted line indicates a vector generated in a current facial image. As shown in FIG. 7, according to the operation of the AU 6

Can be moved clockwise or counterclockwise.

In this state, the curvilinear motion (Mr) of the vertex can be calculated according to the motion of the AU6.

는 AU6의 중점의 움직임 각도이다. 8,

Is the center of gravity of the AU6.

과 움직임 각도

아래와 같은 식에 의해 계산된다. Radius here

And movement angle

Is calculated by the following equation.

The AU-based facial motion information extracted in accordance with the present invention described above, for example, the facial motion size of the AU, the angle, and the facial motion of the apex can be applied in various forms. For example, information of AUs moving according to a facial expression can be extracted and converted into a database, and the information of the facial expression image inputted in real time can be compared with the information in the database to determine the emotion of the current facial expression, have. In addition, the cardiac cardiac information can be extracted from the fine motion of the face portion.

As a result, according to the present invention, it becomes possible to develop a technique of recognizing facial expression recognition using facial motion information based on AU, or cardiac information extraction algorithm. That is, the human facial expression movement is an important factor that can infer the current emotional and cardiac information of the person. The present invention provides a basis for extracting the quantitative change of the AU and applying it.

In order to extract cardiac information from the facial image according to the present invention, the facial motion amount is extracted from the difference value between successive frames by frame, and facial motion data is extracted using the above- do.

Hereinafter, the experiment and verification of the cardiac information extraction method according to the present invention will be described.

1. Subjects

Twenty - four students (12 males, 12 females, ± 1.72) participated in the experiment at Sangmyung University. All subjects had no abnormality or history of cardiovascular system and had enough sleep the day before. In addition, caffeine, smoking, and drinking, which may affect cardiovascular response, were prohibited on the day before the experiment. Before the experiment, all subjects participating in the experiment were explained about the experiment except for the purpose of the study, and then the experiment was carried out and a predetermined amount was paid for the experiment.

2. Experimental Method

The experiment was carried out after allowing the expressionless expression to be maintained without any facial expressions. At this time, no separate stimulation that affects heartbeat and respiration was given, and each individual was able to maintain autonomous respiration. During the experiment, a pulse wave (PPG) sensor was worn and the image of the upper body was measured at the same time. The measurement time was maintained for 3 minutes under autonomous respiration.

3. Analysis and signal processing methods

Pulse wave (PPG) signals were sampled at 500 Hz through the lead-I method. Electrocardiogram signals were obtained by amplifying the signals through an MP100 power supply and a PPG 100C amplifier (Biopac systems Inc., USA) and converting the analog signals to digital signals via NI-DAQ-Pad9205 (National instruments, USA). The pulse wave signal obtained from the sensor detects the peak value of the PPG signal by using the QRS detection algorithm (Detection Algorithm). Peak to Peak Intervals (PPI) were calculated through the difference between the detected peak values and the values of adjacent peaks. The video signal was subjected to signal processing after acquiring data through the method of extracting facial motion (fine motion). Also, for statistical verification between two different signals, the window size was subjected to signal processing under the same condition of 30 seconds. The data extracted after the signal processing were subjected to correlation analysis through SPSS 17.0K to derive the correlation coefficient and the errors such as mean, standard deviation, and root mean square error (RMSE) Data analysis was performed using the analysis method.

4. Method of inferring cardiac information

As shown in FIG. 9, cardiac response data can be inferred using micro movement (MM) of the AU mapped to the face portion (S91). The fine movement is extracted in the above-described process (S27).

In the face of the human body, AUs of various localities are detected. In this case, SD (standard deviation) is used to select the local part of the face having stable and effective information with less motion deviation (S92). When a local region showing stable motion is selected, a BPF (band pass filter) is applied to a band of 0.7-1.3 Hz (S93), and then a peak detection algorithm is applied to find a peak of a heartbeat (S94 ).

RRI (R-peak to R-peak Intervals) was extracted by calculating the interval between peaks and peaks detected, and RRI corrected by the correction coefficients classified after statistical verification was extracted (S95). Based on the corrected RRI, the threshold was set through statistical validation (S96).

And a BPM (Beat Per Minute) signal is extracted based on the corrected fine motion data (S97).

5. Breathing Information Inference Method

FIG. 10 shows a process of extracting respiration information through signal processing for micro-movement.

As in the case of the HR inference method, a fine motion (S101) based on the difference between the current frame and the previous frame is used to extract fine motion data from the tracked AU.

The fine movement is extracted in the above-described process (S27). In the detected AUs, the error of SD (standard deviation) of the motion information was used to select a stable local face of the face with less motion deviation (S102).

When the local region showing stable motion is selected, Band Pass Filter (BPF) is set to 0.7-1. After the filter is applied in the frequency band of 3 Hz (S103), a peak detection algorithm is applied (S104) to find a peak.

The interval between the detected peaks and peaks is calculated and the RRI is extracted. Then, the corrected RRI is extracted through the classification coefficients after the statistical verification (S104).

In the above process, the window size of the data (Data) is set to 30 seconds to collect the reference data for the signal processing for extracting the respiration information, and after the first 30 seconds of data is collected The interval interval is set to 1 second, and data information to be collected is updated using a sliding window technique. Based on the updated data, a signal processing process is performed to extract RSP (Respiration) by using a moving average. The signal processing process is performed by calculating the interval of peaks of the signal, Information (RSA, respiratory sinus arrhythmia signal) is extracted (S106).

6. Results

11 shows the SD analysis result of the motion information.

(T = 39.30244, p <.01) and r> 0.8 (t = 0.39, p <.05), respectively, when the mean values of each motion were classified based on the correlation analysis between the fine data and the PPG signal. (P <.001), r> 0.7 (t = 72. 18763, p <.01) and r> 0.6 (t = 62. 60285, p <.01) were statistically significant It looked.

12 shows a result of a regression analysis according to motion information.

When the correlation between the PPG signal and the fine (MM) signal was 0.9 or more, the range of the cardiac number was divided into upper, middle, and lower (z = -2.503 , p <.05), moderate (z = -5.307, p <.001), and upper and lower (z = -2.345, p <.05) .

X = -0.01293, y = 0.705283 when the regression result is upper (upper), x = 0.008202, y = 0.832221 when the middle (middle) = 0.536307, y = 0.563826.

Table 2 below shows the results of extracted cardiac information.

We compared PPI, BPM, and error between MM (Micro-Movement) signal inferred based on the local motion of the face and sensor-based inferred signal. PPI calculated from PPG (PPG_PPI) was 0.838825, and PPI (MM_BPM) calculated from MM was 0.838924, which showed a difference of 0.000099. The BPM (PPG_BPM) calculated from the PPG was 72.8529096, and the BPM (MM_BPM) calculated from the MM was 72. 84675, which was 0.012311.

Figures 13 and 14 show the results of the data analysis of the 24 subjects and show the results of PPI and BPM, and Figures 15 and 16 show the cases of respiratory sinus arrhythmia (RSA) and peak number.

13, the solid line shows the PPI measured by the sensor, and the dotted line shows the change of the PPI extracted by the present invention. It can be seen from FIG. 13 that the PPI according to the extraction method according to the present invention has a very similar pattern to the PPI of the direct detection method by the sensor.

In FIG. 14, the solid line shows the BPM measured by the sensor, and the dotted line shows the change of the BPM extracted by the present invention. 14, it can be seen that the BPM data according to the extraction method according to the present invention has a very similar pattern to the BPM data directly detected by the sensor.

In FIG. 15, the solid line shows the RSA measured by the sensor and the dotted line compares the changes of the RSA extracted by the present invention. As shown in FIG. 15, the RSA data according to the extraction method according to the present invention has a similar pattern to that of the RSA directly detected by the sensor, although the difference is slight.

In FIG. 16, the solid line indicates the number of peaks measured by the sensor, and the dotted line indicates the change in the number of peaks extracted by the present invention. 16, it can be seen that the peak number data according to the extraction method according to the present invention has a similar pattern to the peak number data directly detected by the sensor.

7. Comparison of Prior Art

Based on the camera-based motion data, the comparison results with the MIT CSAIL study show that the BPM error ratio is 0.014% and the peak count is 16. 89%.

Another previous study compared the mean error, SD error, RMSE error, and correlation of the MIT Media Lab, which is a study using RGB and ICA on the camera base. The correlation coefficients are r = 1, Error = 0.012, SD = 0.06, and RMSE = 0.06.

As described above, the present invention deduced cardiac information using non-contact sensing technology based on fine movement of the muscles of the facial muscles. 3D model-based tracking technology was applied to naturally track the fine movement of the face and extract data from it. AU was selected by mapping the face muscle of Candidate-3 model and FACS. Based on the selected AU, each facial expression and fine motion can be extracted in natural facial tracking state.

And classified by using the SD pattern of the data which is extracted from the AU based data. In the SD pattern, the average motion amount was extracted by setting the window size to 10 seconds in extracting the data of the motion of the AU, and statistical verification was performed by comparing the patterns according to the correlation with the PPG signal. As a result, when the correlation was r> 0.9 (t = 39.30244, p <0.01), r> 0.8 (t = 49.15492, p < , and r> 0.6 (t = 62.60285, p <0.01) were statistically significant. AU of the facial area when r> 0.9 was selected. Data from the selected AUs were classified into upper, middle, and lower parts of the heart rate, and statistical tests were performed.

(Z = -2.503, p <.05), moderate (z = -5.307, p <.000), upper and lower (z = -2.345, p <.019) There were statistically significant differences between the two groups. The regression equation was derived by statistical verification at each step showing significant differences. Regression analysis showed that x = -0.01293 and y = 0.705283 for the upper case and x = 0.008202 and y = 0.832221 for the middle case and x = 0.536307 and y = 0.563826 for the lower case respectively.

Twenty - four male and female college students were measured for autonomic respiration by PPG and MR for about 3 minutes. The measured data were compared with the average of the subjects between the PPG sensor and the microdata, and the PPI, BPM, and BPM errors were analyzed and compared. As a result, PPI (0.838825) and MM (0.838924) showed a difference of 0.000099 in the case of PPI and 0.012332 in the case of BPM with PPG (72. 859096) and MM (72. 846785), respectively.

Correlation analysis of BPM showed that the correlation coefficients were very similar to the heart rate extracted from PPG sensor with r = 1, Error = 0.012, SD = 0.06, and RMSSE = 0.06. In previous studies, the results of correlation analysis, mean error, SD error, root mean square error (RMSE), and peak count (peak number) showed excellent results.

As described above, the present invention confirms that the cardiac information can be measured with excellent results by using the fine detection technology as the non-contact type sensing technology. Especially, although there are many previous studies and similar technologies, it is very difficult to utilize non-contact sensing technology in real life. It is also limited in that it can be measured from the image according to the situation. However, we can confirm that cardiac information can be extracted by using various points of the facial muscles. However, in the present invention, it is necessary to study the lightening of the interlocking module and the more natural tracking technique by applying a tracking module or the like. Therefore, it is expected that cardiac information deduced from the new sensing technique and various facial movement can be applied to real life and related research fields.

These and other embodiments of the present invention have been described and shown in the accompanying drawings. However, it should be understood that these embodiments are only a part of various embodiments. Since various other modifications could occur to those of ordinary skill in the art.

1: Computer system
11: Body
12: Monitor
13: Video camera
14: Keyboard
15: Mouse
20: Subjects

Claims (11)

Mapping a plurality of vertices to a facial image of the subject by masking (matching) a standard model defining a reference vertex for each facial region on the facial image while obtaining a facial image from the subject;
Extracting a coordinate change value between vertices at the same facial region in the facial image;
Continuously extracting facial micro-movement (MM) information including a change value of a centroid of each of a plurality of Action Units (AU) defined by the vertices of the human face;
Selecting fine motion information of AUs having some effective fine motion (MM) information among the plurality of AUs; And
Detecting RRI (R-Peak to R-Peak Intervals) from the selected fine motion information, and extracting cardiac information of the subject using the detected RRI; and extracting cardiac information using facial fine motion. The method according to claim 1,
The facial motion information includes the facial motion size, the area size, the facial motion angle of the polar concept, the facial motion of the vertex (r * Sin (

) Values of the at least one of the at least two of the at least one of the at least two of the at least two of the at least one of the at least two of the at least two of the at least one of the at least two of the at least one at least one group. The method of claim 3,
Wherein the coordinate of the vertex includes X, Y coordinates on the XY plane, and Z coordinate perpendicular to the XY plane. 4. The method according to any one of claims 1 to 3,
Wherein each of the AUs comprises at least three vertices (Vertex). 4. The method according to any one of claims 1 to 3,
Wherein the facial motion information is extracted from a motion (change) in at least one of an XY plane, an XZ plane, and a YZ plane in X, Y, Z coordinate system. The method according to claim 1,
Extracting initial reference data from facial motion information for a predetermined time to obtain facial motion information;
Extracting the initial reference data, extracting the facial motion information at regular time intervals, and extracting the RRI from the facial motion information. The method according to claim 6,
The step of selecting fine motion information of an AU having some effective fine motion (MM) information among a plurality of AUs includes: selecting fine movement of a specific AU based on the standard deviation pattern of the fine motion of the AU , And
Filtering the selected fine motion information in a predetermined frequency range, and detecting a peak from the filtered fine motion information;
Obtaining the RRI average using the peak; And
Detecting a Beat Per Minute (BPM) signal using the RRI; and extracting a heart beat information using a facial fine motion. A system for performing the method according to any one of claims 1 to 3, 6 or 7,
A moving picture camera for acquiring a facial image from the subject;
An image processing unit for processing an image from the video camera;
And a system for extracting the AU facial motion information using the image data from the image processing unit and acquiring cardiac information therefrom. 9. The method of claim 8,
Wherein the AU facial motion information is extracted from a motion (change) in at least one of an XY plane, an XZ plane, and a YZ plane in X, Y, and Z coordinate systems. 10. The method of claim 9,
Wherein each of the AUs includes at least three vertices (Vertex). 9. The method of claim 8,
Wherein the facial motion information is extracted from a motion (change) in at least one of an XY plane, an XZ plane, and a YZ plane in X, Y, and Z coordinate systems.

Images