KR20170121110A - Method and apparatus for extracting Brain information - Google Patents

Abstract

The present invention relates to a method for extracting brain information of a subject to be tested through a pupil size change rate. The method comprises the steps of: acquiring a video from a face of a subject to be tested; extracting a pupil area from the video; extracting a pupil size change rate from an image of the pupil area; and extracting brain information of the subject to be tested from the pupil size change rate.

Description

TECHNICAL FIELD The present invention relates to a method and apparatus for extracting brain information (EEG spectrum)

The present invention relates to a method and apparatus for extracting brain information (EEG spectrum) using a pupil size change rate.

Vital Sign Monitoring (VSM) refers to a technique for acquiring biometric information using a sensor attached to a user's body. Biometric information of the user acquired through the sensor includes pulse, blood pressure, electrocardiogram, body temperature and the like. The biometric information obtained from the user is applied to a variety of industrial fields such as U-healthcare industry, Emotion Marketing, Services, Therapy, security industry, education industry, etc. to create application and added value. In addition, it is possible to provide new products and services with functions and forms completely different from those of existing products through the innovation of value-added products and services in the industry (Jung Byung Joon & Choi Seon Hoon, 2008; 2014).

Conventional bio-signal monitoring technology is an unrealistic sensing method of attaching a sensor to a body in order to acquire a biological response to a user. Therefore, there is a need for non-restraint / non-self-sensing technology that measures biometrics while not being in contact with the human body and not giving any pain and not interfering with the user's activities (The Twins & Park Seung Hoon, 2013). Accordingly, techniques for inferring biometric information of a user using camera technology have been developed. The bioinformatic inference technique using a camera can be regarded as a completely non-constrained / noninvasive biological signal sensing technology. The MIT media lab developed a technique for inferring the heartbeat by analyzing the color information of the face that appears finely according to the blood flowing from the heart to the face (Poh et al ., 2011). The MIT CSAIL lab has also reported a technique for inferring heart beat by micro-movement of the head during blood flow from the heart to the head and PCA analysis of the micro-movement of the head to find the heart-related frequency domain Balakrishnan et al ., 2013).

The noncontact cardiac information reasoning technique reported above does not sufficiently consider cardiac variables with reasonably probable cardiac variables and high actual utilization. In addition, the number of samples used in the analysis was 12 based on the average value of the subjects in the data verification process. Therefore, the reliability of the verification is insufficient and the reliability of the verification is insufficient by verifying the inferred data based on the self-developed ECG sensor. Accordingly, the present invention proposes a methodology for securing an increase in the effective inference parameters of the cardiac information and high accuracy against the cardiac information reasoning technique of the prior art. The non-constrained / non-conscious biometric information reasoning technology developed in the present invention is applied to various industrial fields such as U-healthcare (wellness IT), emotion marketing, services, therapy etc., It is expected to create new value based on services.

Balakrishnan, G., Durand, F., and Guttag, J. (2013). Detecting pulse from head motions in video. In Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on (pp. 3430-3437). IEEE. Poh, M. Z., McDuff, D. J., and Picard, R. W. (2011). Advancements in noncontact, multiparameter physiological measurements using a webcam. Biomedical Engineering, IEEE Transactions on, 58 (1), 7-11. The Twins, Park Seung Hoon. (2013). IT convergence trend in the wellness field. However, (2008). Wireless Body Area Network Technology Trend. Electromagnetic Wave Technology, 19 (3), 35-46. Jong Hye - Sil. (2014). Trends and prospects of healthcare wearable devices. KHIDI Brief, 115.

The present invention discloses a method and apparatus for extracting frequency information of each brain point from a pupil image.

Method according to the invention:

Acquiring a moving picture on the face of the subject;

Extracting a pupil region from the moving image;

Extracting a pupil size change rate from the image of the pupil region;

And extracting brain information of the subject from the pupil size change rate.

According to an embodiment of the present invention, the pupil size change rate may be filtered by frequency bands, and a 1/100 harmonic frequency may be applied to extract data corresponding to brain information of the corresponding frequency band.

According to another embodiment of the present invention, the power spectral density can be extracted by FFT analysis on the data filtered by the frequency bands, and the extracted power can be summarized.

According to another embodiment of the present invention, the power amount (%) of each frequency band can be calculated by setting the power value obtained by FFT analysis of the data filtered by the frequency band as total power.

A brain information extracting apparatus using the pupil size change rate according to the present invention performing the above method comprises: a moving picture camera for photographing the face of the subject;

And an analysis system for extracting a pupil region from the moving image, extracting a pupil size change rate using the extracted pupil size, and extracting data corresponding to brain information of the subject.

The present invention proposes a method and system for extracting frequency information (FP1 - low beta, mid beta, SMR, F3 - beta, F8 - high beta, C4 - mu and P4 - gamma) of a brain from a pupil image. Therefore, we analyzed the accuracy between the data obtained from the EEG sensor and the data deduced from the pupil data, and confirmed the high correlation between the two signals and the low error rate. In addition, we have developed a system that can infer brain frequency information by acquiring pupil images in real time based on the above algorithm. The non-constrained / non-conscious brain information reasoning technology developed in the present invention is applied to various industrial fields such as U-healthcare (wellness IT), emotion marketing, services, therapy etc., It is expected to create new value based on services.

Figure 1 shows an EEG (Electroencephalogram) measurement point according to International 10-20 method.
FIG. 2 illustrates a pupil image processing process according to the present invention.
3 illustrates an EEG signal processing process according to the present invention.
FIG. 4 shows the result of comparing the EEG spectrum (low beta in FP1 area) based on the pupil and EEG signals (Participants 7).
FIG. 5 shows the result of the signal comparison between the pupil and EEG spectrum (mid beta in FP1 area) (Participants 7).
FIG. 6 shows the results of a comparison of signals of the EEG spectrum (SMR in FP1 area) based on pupil and EEG (Participants 7).
FIG. 7 shows a comparison result of the EEG spectrum (beta in F3 area) based on pupil and brain wave (Participants 7).
FIG. 8 shows the results of the EEG spectrum (high beta in F8 area) based on pupil and brain wave (Participants 7)
Figure 9 shows the results of the pupil and EEG-based EEG spectrum comparison (Participants 7).
FIG. 10 shows the result of comparing the EEG spectrum (gamma in P4 area) based on pupil and EEG signals (Participants 7).
FIG. 11 shows the signal correlation and error rate comparison results of the EEG spectrum based on the pupil and EEG.
Figure 12 illustrates an interface screen of a pupil-based brain response reasoning system.

Hereinafter, a PC-based system including a method of extracting frequency-specific frequency information of brain waves from a pupil image and an image camera and an analysis system according to the present invention will be described with reference to the accompanying drawings.

1. Subjects

In the experiments of the present invention, 15 subjects (7 male and 7 female, mean age: 28.2 ± 3.24) participated in the experiment. The subjects were those who had no history or history of central and autonomic nervous system and visual function. Prior to participating in the experiment, drinking, smoking, and caffeine, which could affect the nervous system, were limited and enough sleep was allowed on the day before the experiment to minimize fatigue on the day of the experiment. The subjects were informed of the outline of the experiment except for the purpose of the study, and then they received the consent of volunteers to participate voluntarily. I paid a certain amount of money to participate in the experiment and increased my intention to participate in the experiment.

2. Experimental Procedure

The subject was allowed to sit on a comfortable chair and the distance between the subject and the screen was 1 m. A gray bar was stared as a reference stimuli presented on the screen for 3 minutes in the absence of motion. The electrocardiogram (ECG, electrocardiogram) and pupil image were simultaneously measured while the subject stared at the screen. Illumination of the experimental environment and stimulation that could affect the pupil measurement was controlled by the same conditions for all subjects.

3. Data collection and signal processing

As shown in FIG. 1, the brain waves are generated by using the global electrode arrangement method '10 -20 'based on the FP1, FP2, F3, Fz, F4, F7, F8, C3, Cz, C4, T3, T4, T5, T6, P3, Pz, P4, O1, and O2 were measured. The AFz point was set to ground and both ears were referenced (Ear Reference, A1 & A2). The brain waves were acquired at a 500 Hz sampling rate using a Mitsar-EEG 202 Machine (Mitsar Ltd., Russia). The DC level was set at 0 - 150 Hz and acquired at 300 mV. Montage for EEG acquisition was set to Monopolar electrode montages. The collected signals were processed using Labview 2010 software (National Instrument Inc., USA). The pupil images were acquired at 30 frames / second using a point gray camera (FL3-GE-50S5M-C, Canada) and the resolution of the images was 960x400.

In order to extract the pupil region from the acquired infrared image, we used OpenCV for binarization through gray scale and threshold. The pupil region was tracked using the contour detection method in the binarized image from which the region other than the pupil was removed. In the image processing program, rect.width, which is the radius of the pupil, is applied to the circularity formula only when it meets the condition of Equation 1 below to measure the blinking of the pupil, which affects the pupil size, And the pupil size was extracted.

That is, when the ratio of the rect (height) to the height (rect.width) of the pupil is greater than or equal to 0.5, the blinking is determined. In order to compare with EEG data, band pass filter (BPF) was applied to each frequency band. Since the pupil size change rate data can be used to confirm the frequency information up to 0.5 Hz with the data sampled at 1 Hz, the concept of harmonic frequency of 1/100 ratio is applied to compare with the EEG frequency. The harmonic frequency information of 1/100 is represented by Delta (0.01-0.04 Hz), Theta (0.04-0.08 Hz), Alpha (0.08-0.13 Hz), Beta (0.13-0.30 Hz), Gamma (0.30-0.50 Hz) 0.10-0.15 Hz), Mid Beta (0.15-0.18 Hz), High Beta (0.20-0.30 Hz), Mu (0.9-0.12 Hz), Alpha (0.08-0.09 Hz) ) And SMR (0.12 - 0.15 Hz) band, respectively. Data processed by each frequency band is extracted by Power Spectrum Density (FFT) analysis and summarized.

In this case, the power of each extracted frequency band is calculated by using the power obtained by FFT processing of the BPF-processed pupil size data at 0.01-0.5 Hz as total power, and the power amount (%) of each frequency band is calculated. The power (%) of the frequency bands of Delta, Theta, Alpha, Beta, Gamma, Slow alpha, Fast alpha, Low beta, Mid beta, High beta, SMR, , Alpha, Beta, Gamma, Slow alpha, Fast alpha, Low beta, Mid beta, High beta, SMR, Mu. Detailed signal processing is shown in FIG.

EEG data were obtained by filtering the noise components of the remaining bands excluding the EEG frequency band required for analysis from Delta (1 - 4 Hz) to Gamma (30 - 50 Hz) through BPF (low cut: 1 Hz, (See Equation 2).

Fast Fourier transform (FFT) analysis was performed on the EEG signal processed by BPF (low cut: 1 Hz, high cut: 50 Hz) (see Equation 3). Delta (1 - 4 Hz), Theta (4 - 8 Hz), Alpha (8 - 13 Hz), Beta (13 - 30 Hz) and Gamma (30 - 50 Hz), Slow alpha (8-9 Hz), Fast Alpha (10-12 Hz), Low Beta (12-15 Hz), Mid Beta (15-18 Hz) (9 - 12 Hz) and SMR (12 - 15 Hz), respectively. The power value extracted for each frequency band is calculated by the ratio (Percentage,%) that the power value per frequency band is compared with Total Power (1 - 50 Hz). This was compared with the results of inferring through pupil response. Detailed signal processing is shown in FIG.

4. Experimental results

We compared the EEG spectrum signal deduced based on the pupil size change rate and the EEG spectrum data extracted from the sensor based EEG data. 1, there is a high correlation in the low beta, mid beta and SMR bands at FP1 (low beta - r = .684, p <.05 / mid beta - r = .576, p < (low beta: 0.26%, mid beta: 0.19%, SMR: 0.26%). The detailed results are shown in Figs. 4-6 (Participants 7).

At the F3 point, the correlation was high in the beta band (r = .624, p <.05). It was confirmed that the deviation between the inferred data is small (error rate: 0.08%). The detailed results are shown in Fig. 7 (Participants 7).

At the F8 point, the correlation was high (r = .582, p <.05) in the high beta band and the deviation between the inferred data was small (error rate: 0.1%). The detailed results are shown in FIG. 8 (Participants 7).

At the C4 point, the correlation between the inferred data was high (r = .576, p <.05) and the error rate was 0.07%. The detailed results are shown in Fig. 9 (Participants 7).

At the P4 point, we found that the correlation between the inferred data is high (2.47%) with high correlation in the gamma band (r = .639, p <.05). Detailed results are shown in Fig. 10 (Participants 7).

Correlation and error rate of 160 data samples were calculated by sliding window (Window size: 30s, Resolution 1s) for each subject. Correlation analysis was performed by Pearson correlation analysis and the error rate was calculated by the following equation (5).

The results of the accuracy comparison of EEG spectrum data of 15 subjects are shown in FIG. The accuracy of the extracted EEG spectrum data based on the pupil and electrocardiogram was found to be strong correlations (FP1 - low beta: r = 0.613 ± 0.048, p <.05, mid beta: r = 0.599 ± 0.036, p < .05, SMR: r = 0.599 ± 0.029, p <.05, F3 - beta: r = 0.632 ± 0.027, p <.05, F8 - high beta: r = 0.610 ± 0.033, p < The mean error rate was (FP1 - low beta: 0.173 ± 0.079 (%), mid beta: 0.181 (p = Beta: 0.682 ± 1.593 (%), F8 - high beta: 0.492 ± 1.131 (%), C4-mu: 1.773 ± 5.335 (%), P4 - gamma: 1.646 ± 1.938 (%)).

A system capable of inferring brain frequency information based on the pupil image acquired in real time based on the above brain information inference algorithm was developed and the developed system screen is shown in FIG. In Fig. 12, (1) shows the input image, (2) shows the binarized image, (3) shows the pupil tracking image, and (4) shows the power amount inference results per brain point.

5. Conclusion

The method and system for extracting frequency information (FP1 - low beta, mid beta, SMR, F3 - beta, F8 - high beta, C4 - mu and P4 - gamma) Respectively. For this, we analyzed the accuracy between the data obtained from EEG sensors and the data deduced from the pupil data, and confirmed the high correlation between the two signals and the low error rate. In addition, we have developed a system that can infer brain frequency information by acquiring pupil images in real time based on the above algorithm. The non-constrained / non-conscious brain information reasoning technology developed in the present invention is applied to various industrial fields such as U-healthcare (wellness IT), emotion marketing, services, therapy etc., It is expected to create new value based on services.

In the foregoing, exemplary embodiments have been described and shown in the accompanying drawings to facilitate understanding of the present invention. It should be understood, however, that such embodiments are merely illustrative of the present invention and not limiting thereof. And it is to be understood that the invention is not limited to the details shown and described. Since various other modifications may occur to those of ordinary skill in the art.

Claims (5)

Acquiring a moving picture on the face of the subject;
Extracting a pupil region from the moving image;
Extracting a pupil size change rate from the image of the pupil region; And
And extracting the brain information of the subject from the pupil size change rate. The method according to claim 1,
Wherein the pupil size change rate is filtered for each frequency band, and a 1/100 harmonic frequency is applied to extract the data corresponding to the brain information of the corresponding frequency band. 3. The method of claim 2,
Extracting Power Spectrum Density through FFT analysis on the data filtered by the frequency bands, and summarizing the extracted powers. The method of claim 3,
And calculating a power amount (%) of each frequency band by using a power value obtained by FFT analysis of the data filtered by the frequency bands as total power. A brain information extracting apparatus for performing the method according to any one of claims 1 to 4,
A moving picture camera for photographing the face of the subject;
And an analysis system for extracting a pupil region from the moving image, extracting pupil size change rate using the extracted pupil region, and extracting data corresponding to brain information of a subject.

Images