KR20170120546A - Method and system for extracting Heart Information of Frequency domain by using pupil size variation - Google Patents

Abstract

The method of the present invention comprises: photographing a pupil of a subject; Extracting a change rate of a pupil size; And extracting information on the time domain of the heart using the rate of change.

Description

FIELD OF THE INVENTION [0001] The present invention relates to a method and apparatus for extracting cardiac frequency domain information using a change in pupil size,

The present invention relates to a method and apparatus for extracting cardiac information in a wavenumber region using a pupil size change rate.

Vital Sign Monitoring (VSM) refers to a technique for acquiring biometric information using a sensor attached to a user's body. Biometric information of the user acquired through the sensor includes pulse, blood pressure, electrocardiogram, body temperature and the like. The biometric information obtained from the user is applied to a variety of industrial fields such as U-healthcare industry, Emotion Marketing, Services, Therapy, security industry, education industry, etc. to create application and added value. In addition, it is possible to provide new products and services with functions and forms completely different from those of existing products through the innovation of value-added products and services in the industry (Jung Byung Joon & Choi Seon Hoon, 2008; 2014).

Conventional bio-signal monitoring technology is an unrealistic sensing method of attaching a sensor to a body in order to acquire a biological response to a user. Therefore, there is a need for non-restraint / non-self-sensing technology that measures biometrics while not being in contact with the human body and not giving any pain and not interfering with the user's activities (The Twins & Park Seung Hoon, 2013). Accordingly, techniques for inferring biometric information of a user using camera technology have been developed. The bioinformatic inference technique using a camera can be regarded as a completely non-constrained / noninvasive biological signal sensing technology. The MIT media lab developed a technique for inferring the heartbeat by analyzing the color information of the face that appears finely according to the blood flowing from the heart to the face (Poh et al ., 2011). The MIT CSAIL lab has also reported a technique for inferring heart beat by micro-movement of the head during blood flow from the heart to the head and PCA analysis of the micro-movement of the head to find the heart-related frequency domain Balakrishnan et al ., 2013).

The noncontact cardiac information reasoning technique reported above does not sufficiently consider cardiac variables with reasonably probable cardiac variables and high actual utilization. In addition, the number of samples used in the analysis was 12 based on the average value of the subjects in the data verification process. Therefore, the reliability of the verification is insufficient and the reliability of the verification is insufficient by verifying the inferred data based on the self-developed ECG sensor. Accordingly, the present invention proposes a methodology for securing an increase in the effective inference parameters of the cardiac information and high accuracy against the cardiac information reasoning technique of the prior art. The non-constrained / non-conscious biometric information reasoning technology developed in the present invention is applied to various industrial fields such as U-healthcare (wellness IT), emotion marketing, services, therapy etc., It is expected to create new value based on services.

Balakrishnan, G., Durand, F., and Guttag, J. (2013). Detecting pulse from head motions in video. In Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on (pp. 3430-3437). IEEE. Pan J., Tompkins W. J., 1985. A real-time QRS detection algorithm. IEEE Trans. Biomed. Eng., 32, 230-236. Poh, M. Z., McDuff, D. J., and Picard, R. W. (2011). Advancements in noncontact, multiparameter physiological measurements using a webcam. Biomedical Engineering, IEEE Transactions on, 58 (1), 7-11. The Twins, Park Seung Hoon. (2013). IT convergence trend in the wellness field. However, (2008). Wireless Body Area Network Technology Trend. Electromagnetic Wave Technology, 19 (3), 35-46. Jong Hye - Sil. (2014). Trends and prospects of healthcare wearable devices. KHIDI Brief, 115.

The present invention provides a method and apparatus for extracting cardiac information in a wavenumber region using a change in pupil size.

A method for extracting frequency-domain information of a heart using pupil size change according to the present invention:

Photographing a pupil of a subject;

Extracting a change rate of a pupil size;

And extracting time-domain information of the heart using the rate of change.

According to an embodiment of the present invention, pupil flicker can be determined according to the detected pupil size while extracting the pupil size change data.

According to an embodiment of the present invention, the power of the resampled data can be obtained by resampling the data of the pupil size to a predetermined frequency.

According to an embodiment of the present invention, the resampled data may be classified into a plurality of bands by BPF (Band Pass Filter), and the amount of power may be calculated by analyzing FFT of data of each band.

According to an embodiment of the present invention, the plurality of bands includes a very low frequency band (0.0033 - 0.04 Hz), a low frequency band (0.04 - 0.15 Hz), a high frequency band (0.15 - 0.04 Hz).

An apparatus for extracting frequency domain information of a heart based on a pupil size change by performing the methods of the present invention,

A camera for photographing the pupil of the subject; And

And an analyzer for extracting the size change data of the pupil and extracting information on a frequency region of the heart by the method according to the first or second aspect.

According to the present invention, the accuracy between the data obtained from the electrocardiogram sensor and the data deduced from the pupil data was analyzed, and high correlation between the two signals and a low error rate were confirmed. In addition, we have developed a system that can infer heart time information by acquiring pupil images in real time based on the above algorithm. The unconstrained / unacknowledged cardiac information reasoning technology developed in the present invention is applied to various industrial fields such as U-healthcare (Wellness IT), Emotion Marketing, Services, Therapy etc., It is expected to create new value based on services.

1 shows an electrocardiogram measuring point according to the present invention.
FIG. 2 shows a process of processing a pupil image signal according to the present invention.
FIG. 3 shows a process of ECG signal processing according to the present invention.
FIG. 4 shows the result of comparing the HRV spectrum (VLF power) signals based on the pupil and ECG (Participants 6).
FIG. 5 shows the result of comparing the HRV spectrum (LF power) signal based on the pupil and ECG (Participants 6).
FIG. 6 shows the result of comparing the HRV spectrum (HF power) signals based on the pupil and electrocardiogram (Participants 6).
FIG. 7 shows the result of comparing the HRV spectrum (VLF / HF ratio) signals based on the pupil and electrocardiogram (Participants 6).
FIG. 8 shows the result of comparing the HRV spectrum (LF / HF ratio) signals based on the pupil and electrocardiogram (Participants 6).
FIG. 9 shows the result of comparing the pupil and ECG-based normalized HRV ( ln VLF & ln HF) data pattern (Participants 6).
FIG. 10 shows the result of comparing the pupil and ECG-based normalized HRV ( ln LF & ln HF) data pattern (Participants 6).
Figure 11 shows the results of verifying the HRV spectrum (VLF power) signal accuracy based on the pupil and electrocardiogram.
Figure 12 shows the results of verifying the HRV spectrum (LF power) signal accuracy based on the pupil and electrocardiogram.
Fig. 13 shows the result of verifying the HRV spectrum (HF power) signal accuracy based on the pupil and electrocardiogram.
Figure 14 shows the results of verifying the HRV spectrum (VLF / HF ratio) signal accuracy based on the pupil and electrocardiogram.
Fig. 15 shows the result of verifying the HRV spectrum (LF / HF ratio) signal accuracy based on the pupil and electrocardiogram.
FIG. 16 shows the results of verifying the normalized HRV ( ln VLF) signal accuracy based on the pupil and electrocardiogram.
17 shows the results of verifying the accuracy of the normalized HRV ( ln LF) signal based on the pupil and electrocardiogram.
FIG. 18 shows the results of verifying the normalized HRV ( ln HLF) signal accuracy based on the pupil and electrocardiogram.
Figure 19 illustrates an interface screen of a system according to the present invention.

Hereinafter, with reference to the accompanying drawings, a method for extracting information of a cardiac frequency region using a pupil size change rate according to the present invention and an embodiment of the apparatus will be described in detail.

An apparatus according to the present invention may include a software-based analysis unit for extracting frequency-domain information of a heart from a camera for photographing a pupil or a face of a subject and image data therefrom as a PC-based apparatus including software and hardware.

1. Subjects

In the experiments of the present invention, 15 subjects (7 male and 7 female, mean age: 28.2 ± 3.24) participated in the experiment. The subjects were those who had no history or history of central and autonomic nervous system and visual function. Prior to participating in the experiment, drinking, smoking, and caffeine, which could affect the nervous system, were limited and enough sleep was allowed on the day before the experiment to minimize fatigue on the day of the experiment. The subject was informed of the outline of the experiment except for the research purpose of the present invention, and then the subject consent was given to the voluntary participation intention. I paid a certain amount of money to participate in the experiment and increased my intention to participate in the experiment.

2. Experimental Procedure

The subject was allowed to sit on a comfortable chair and the distance between the subject and the screen was 1 m. A gray bar on the screen was stared as a reference stimuli presented on the screen for 3 minutes in the absence of motion. Electrocardiogram (ECG) and pupil images were measured and photographed at the same time while the subject gazed at the screen. Illumination of the experimental environment and stimulation that could affect the pupil measurement was controlled by the same conditions for all subjects.

3. Data collection and signal processing

The electrocardiogram measured one channel data through the standard limb induction method (lead I). A detailed ECG measurement method is shown in Fig. Electrocardiograms were amplified using an ECG100C amplifier (Biopac system Inc., USA) and signals were collected at 500 Hz using a NI-DAQ-Pad9205 (National Instrument Inc., USA). The collected signals were processed using Labview 2010 software (National Instrument Inc., USA). The pupil images were acquired at 30 frames / second using a point gray camera (FL3-GE-50S5M-C, Canada) and the resolution of the images was 960x400.

In order to extract the pupil region from the acquired infrared image, we performed the binarization through gray scale and threshold using OpenCV. The pupil region was tracked using the contour detection method in the binarized image from which the region other than the pupil was removed. In the image processing program, only the pupil radius, which is the radius of the pupil, satisfies the condition of <Equation 1> below, which is used to measure the blinking of the pupil in the pupil region stored as "Rect" . width were applied to the equation of circumference to extract the pupil size continuously.

That is, when the ratio of the rect (height) to the height (rect.width) of the pupil is greater than or equal to 0.5, the blinking is determined.

The extracted pupil size data was re-sampled at 1 Hz and the pupil size data re-sampled at 1 Hz had a very low frequency band (0.0033 - 0.04 Hz) and a low frequency (LF) band LF, and HF bands, respectively, through a Fast Fourier Transform (FFT) analysis on a band pass filter (BPF) for each frequency band of the HF band (0.04-0.15 Hz) and the HF band (0.15-0.04 Hz) The amount of star power was calculated.

At this time, the power amount (%) of each frequency band was calculated by taking the power value obtained by FFT processing of the BPF-processed pupil size data at 0.0033-0.4 Hz as the total power. The ratio of the VLF, LF, HF, VLF / HF ratio and LF / HF ratio of the extracted pupil to the power ratio (% ratio, and LF / HF ratio. In addition, VFL, LF, and HF power were extracted from the natural log (Nomalized HRV) ( ln VLF, ln LF, ln HF) data. Detailed signal processing is shown in FIG.

The electrocardiogram data acquired by the sensor detected the R-peak of the ECG signal using the QRS detection algorithm (Pan & Tompkins, 1985). The detected R-peaks were calculated by RRI (R-peak to R-peak Intervals) through the difference between adjacent R-peaks.

RRI data was converted to time-series data by re-sampling (2 Hz) and HRV (heart rate variability) spectrum data was extracted through FFT analysis. Extracted HRV spectral data were obtained by extracting power values of VFL (0.0033-0.04 Hz), LF (0.04-0.15 Hz) and HF (0.15-0.4 Hz) and comparing the total power (0.0033-0.4 Hz) (%) Was calculated. In addition, VFL, LF, and HF Power extract naturalized HRV ( ln VLF, ln LF, ln HF) data by taking a natural log. Detailed signal processing is shown in FIG.

The two data compares the accuracy of the signal with the correlation analysis and mean error and error rate (%).

4. Experimental results

HRV spectral signal deduced based on pupil size change ratio and HRV spectral data extracted from sensor based ECG data were compared. Samples of the two signals of the participants (Participants 6) are shown in FIGS. It was confirmed that the patterns of the two signals are similar and the deviation between data is small.

Also, we compared the normalized HRV data deduced from the pupil size change rate and the normalized HRV data extracted from the sensor based ECG data. Samples of the two data of the participants (Participants 6) are shown in FIGS. It was confirmed that the patterns of the two data are similar and the deviation between the data is small.

Data of 15 subjects were calculated by correlating 160 data samples, mean error and error rate through a sliding window (Window size: 30s, Resolution 1s) for each subject. Respectively. Correlation analysis was performed using Pearson's correlation analysis. The mean error was calculated by the mean of the difference between the two data, and the error rate was calculated using Equation 6.

The results of the accuracy comparison of the 15 HRV spectral data of the subjects are shown in Fig. The accuracy of the extracted HRV spectra (VLF power) data based on the pupil and electrocardiogram was confirmed to be strong correlation (r = 0.765 ± 0.055, p <.001) and the mean error was 0.784 ± 0.277 , And the average error rate was 0.956 ± 0.332 (%).

The results of the accuracy comparison of the HRV spectral data (LF power) of 15 subjects are shown in FIG. As a result of comparing the accuracy of extracted HRV spectral data (LF power) based on pupil and electrocardiogram, a strong positive correlation (r = 0.715 ± 0.058, p <.001) was confirmed and mean error was 0.648 ± 0.216 , And the average error rate was 4.459 ± 1.298 (%).

The accuracy comparison result of the HRV spectrum (HF power) data of 15 subjects is shown in FIG. As a result of comparing the accuracy of extracted HRV spectrum (HF power) data based on pupil and electrocardiogram, a strong positive correlation (r = 0.715 ± 0.037, p <.001) was confirmed and an average error was found to be 0.697 ± 0.289 , And the average error rate was 5.610 ± 2.564 (%).

The results of the accuracy comparison of the HRV spectrum (VLF / HF ratio) data of 15 subjects are shown in Fig. The accuracy of the extracted HRV spectrum (VLF / HF ratio) data based on the pupil and electrocardiogram was confirmed to be strong correlations (r = 0.724 ± 0.040, p <.001) and the mean error was 4.219 ± 2.318 And the average error rate was 5.602 ± 2.848 (%).

The results of the accuracy comparison of the HRV spectrum (LF / HF ratio) data of 15 subjects are shown in FIG. The accuracy of the extracted HRV spectra (LF / HF ratio) data based on the pupil and electrocardiogram was found to be strongly correlated (r = 0.724 ± 0.066, p <.001) and the mean error was 0.841 ± 0.404 And the average error rate was 2.269 ± 1.362 (%).

The accuracy comparison result of the normalized HRV ( ln VLF) data of 15 subjects is shown in Fig. The accuracy of the extracted normalized HRV ( ln VLF) data based on the pupil and electrocardiogram was verified with a strong positive correlation (r = 0.765 ± 0.055, p <.001) and an average error of 0.021 ± 0.013 , And the average error rate was 0.496 ± 0.305 (%).

The results of the accuracy comparison of the normalized HRV ( ln LF) data of 15 subjects are shown in Fig. The accuracy of the normalized HRV ( ln LF) data extracted from the pupil and electrocardiogram was verified with a strong positive correlation (r = 0.715 ± 0.058, p <.001) and the mean error was 0.024 ± 0.014 And the average error rate was 1.103 ± 0.650 (%).

The results of the accuracy comparison of the normalized HRV ( ln HF) data of 15 subjects are shown in Fig. The accuracy of the normalized HRV ( ln HF) data extracted from the pupil and electrocardiogram was verified with a strong positive correlation (r = 0.735 ± 0.037, p <.001) and an average error of 0.008 ± 0.007 And the average error rate was 0.639 ± 0.529 (%).

A system has been developed that can infer cardiac time information based on a pupil image acquired in real time based on the cardiac information reasoning algorithm described above. The developed system screen is shown in FIG. In FIG. 19, reference numeral 1 denotes an input image, reference numeral 2 denotes a binarized image, reference numeral 3 denotes a pupil tracking image, reference numeral 4 denotes a heartbeat reasoning signal, reference numeral 5 denotes an HRV spectrum, As a result of inference, (6) represents the results of normalized HRV inference.

In the foregoing, exemplary embodiments have been described and shown in the accompanying drawings to facilitate understanding of the present invention. It should be understood, however, that such embodiments are merely illustrative of the present invention and not limiting thereof. And it is to be understood that the invention is not limited to the details shown and described. Since various other modifications may occur to those of ordinary skill in the art.

Claims (9)

Photographing a pupil of a subject;
Extracting pupil size change data;
Extracting frequency domain information of the heart from the change data; and extracting frequency domain information of the heart using the change of the pupil size. The method according to claim 1,
And extracting the pupil size change data and determining a pupil flicker according to the detected pupil size. 3. The method according to claim 1 or 2,
The method comprising the steps of: resampling the pupil size data to a predetermined frequency and obtaining a power amount of the resampled data; and extracting the frequency domain information of the heart using the pupil size change. The method of claim 3,
Wherein the resampled data is classified into a plurality of bands by a band pass filter (BPF), and the amount of power is calculated by analyzing data of each band by FFT. A method for extracting area information. 5. The method of claim 4,
The plurality of bands include a very low frequency band (0.0033 - 0.04 Hz), a low frequency band (0.04 - 0.15 Hz), and a high frequency band (0.15 - 0.04 Hz) And extracting the frequency domain information of the heart using the change in the pupil size. An apparatus for extracting frequency-domain information of a heart using a change in a pupil size by performing the method according to claim 1 or 2,
A camera for photographing the pupil of the subject; And
And analyzing means for extracting the size change data of the pupil and extracting information on a frequency region of the heart by the method according to the first or second aspect of the present invention, Extracting device. The method according to claim 6,
Wherein the analysis unit resamples the pupil size data to a predetermined frequency and obtains a power amount of the resampled data, and extracts the frequency domain information of the heart using the pupil size change. 8. The method of claim 7,
Wherein the analysis unit classifies the resampled data into a plurality of bands by a band pass filter (BPF), and calculates the amount of power by analyzing data of each band by FFT. An apparatus for extracting frequency domain information of a heart. 9. The method of claim 8,
The analyzer analyzes the resampled data by a band pass filter (BPF) in a very low frequency band (0.0033-0.04 Hz), a low frequency band (0.04-0.15 Hz), a high frequency band (HF band) 0.15 - 0.04 Hz). The apparatus for extracting the frequency domain information of the heart using the pupil size change.

Images