CN114998229A

CN114998229A - Non-contact sleep monitoring method based on deep learning and multi-parameter fusion

Info

Publication number: CN114998229A
Application number: CN202210561402.3A
Authority: CN
Inventors: 张静; 晏博赟; 贺涛; 杜晓辉; 王祥舟; 孙海鑫; 刘娟秀; 刘霖; 刘永
Original assignee: University of Electronic Science and Technology of China
Current assignee: University of Electronic Science and Technology of China
Priority date: 2022-05-23
Filing date: 2022-05-23
Publication date: 2022-09-02
Anticipated expiration: 2042-05-23
Also published as: CN114998229B

Abstract

The invention discloses a non-contact sleep monitoring system based on deep learning and multi-parameter fusion, and belongs to the field of image processing and deep learning. The system firstly segments an acquired sleep video image, then builds a deep convolutional neural network for extracting and amplifying physiological signals, amplifies a heart rate signal of a forehead area and eye movement frequency of an eye area by setting different amplification factors of the network to obtain a forehead area video image for amplifying the heart rate signal and an eye area video image after amplifying the eye movement frequency, extracts corresponding frequency spectrum by utilizing fast Fourier transform, and finds out frequency corresponding to a frequency spectrum peak value as the monitored heart rate signal and eye movement frequency. And for the three-position body video images, building a deep learning-based sleeping posture monitoring neural network structure, inputting the sleeping posture characteristics extracted from the network into a full connection layer for six classification, wherein the classification results correspond to six sleeping postures of supine, prone, left-side straight lying, left-side lying, right-side straight lying and right-side lying, and counting the turn-over times by switching different sleeping postures of the six sleeping postures. And finally, comprehensively evaluating the sleep quality by integrating the monitored physiological signals. The invention has the characteristics of high comfort, multi-parameter fusion and high automation, and realizes the monitoring of physiological parameters such as heart rate, respiratory rate, eye movement rate, sleeping posture, turnover frequency and the like in a non-contact manner.

Description

Non-contact sleep monitoring method based on deep learning and multi-parameter fusion

Technical Field

The invention belongs to the field of image processing and the field of deep learning, and particularly relates to a non-contact sleep monitoring system for realizing multi-parameter fusion by combining a video image processing technology and a deep convolutional network.

Technical Field

In the sleeping process, a series of functions of the brain, muscles, eyes, the heart, breath and the like of a human body can change, and the judgment on the sleeping quality of the human body can be promoted by monitoring the changes. Sleep disorders generally refer to abnormalities in the quality or quantity of sleep or the occurrence of certain clinical symptoms during sleep, such as decreased or excessive sleep, sleep disordered breathing, disorders of rapid eye movement sleep behavior, and the like. Medical science proves that people with sleep disorder for a long time can induce various diseases, so that the timely diagnosis and treatment of the sleep disorder have important significance on human health.

Polysomnography is a golden method called sleep disorder diagnosis and treatment, and mainly monitors physiological signals of channels such as electroencephalograms, electrocardiograms, electrooculogram, oronasal airflow flux, blood oxygen saturation and the like of patients, and diagnoses according to the collected signals. The utility model discloses a measurement of sleep quality, including the person who is surveyed, need install multiple sensor on the person's body when utilizing polysomnography to monitor, bring very big discomfort for the person who is surveyed, in addition, even it monitors many parameters, the doctor still can be according to the past medical history of the person who is surveyed and the subjective impression during monitoring as one of the foundation of aassessment, and the interpretation result has stronger subjectivity. With the development of deep learning, many miniaturized sleep monitoring devices, such as intelligent pillows, mattresses, bracelets, etc., have appeared. The intelligent pillow and the mattress monitor the pressure change in the sleeping process through the pressure sensor, and the turn-over times of the tested person are counted. The intelligent bracelet can monitor the heart rate during sleep when worn. Although the existing sleep monitoring equipment reduces the uncomfortable feeling of wearing a sensor in the monitoring process, the existing sleep monitoring equipment can only monitor a single physiological parameter, so that the evaluation result of the sleep quality is inaccurate and incomplete.

Aiming at the problems in sleep monitoring, a non-contact sleep monitoring system based on deep learning and multi-parameter fusion is designed. For the heart rate and the eye movement frequency, a micro physiological signal is extracted from a sleep video through a deep convolutional neural network and amplified, meanwhile, the generation of artifacts is inhibited, the effective physiological signal amplification is realized, and finally, the amplified physiological signal is subjected to spectrum analysis. And for the sleeping postures and the turnover times, automatically extracting the sleeping posture characteristics of the video frames acquired by the three-position camera by using a convolutional neural network, and carrying out six classifications on the sleeping posture characteristics, wherein the turnover times in the sleeping process can be monitored according to the switching among different sleeping postures.

Disclosure of Invention

The invention designs a non-contact sleep monitoring system based on deep learning and multi-parameter fusion aiming at the problems that the contact monitoring of a polysomnogram brings discomfort and subjectivity of manual interpretation to a tested person and the singleness of monitoring physiological parameters of other sleep monitoring equipment and the like so as to realize the non-contact monitoring of multiple physiological parameters such as heart rate, eye movement frequency, sleeping posture, turning-over times and the like.

The technical scheme of the invention is a non-contact sleep monitoring method based on deep learning and multi-parameter fusion, which comprises the following steps:

step 1: building a sleep monitoring platform, and respectively placing three cameras at the upper, left and right positions of a tester body to obtain video images of the tester in the sleeping process;

step 2: performing image segmentation on the video image obtained by the camera positioned above the body of the tester in the step 1 to obtain a forehead area video image and an eye area video image of the tester;

and step 3: constructing a deep convolutional neural network for extracting and amplifying physiological signals, and extracting and amplifying tiny physiological signals in a video by using the deep convolutional neural network;

and 4, step 4: respectively inputting the forehead area video image and the eye area video image obtained in the step 2 into the deep convolutional neural network built in the step 3, respectively extracting and amplifying a heart rate signal of the forehead area and an eye movement signal of the eye area, and outputting a forehead area video image after amplifying the heart rate signal and an eye area video image after amplifying the eye movement frequency;

and 5: carrying out RGB three-channel separation on each frame of image in the forehead area video image obtained in the step 4 after the heart rate signal is amplified, averaging pixel points in R, G, B three channels, and then carrying out time sequence stacking to obtain a pulse wave signal;

step 6: performing fast Fourier transform on the pulse wave signals obtained in the step 5 to obtain a time series frequency spectrum of the human pulse waves;

and 7: performing spectrum analysis on the time series spectrum obtained in the step 6, and selecting the frequency corresponding to the peak value of the spectrum as a heart rate monitoring result;

and 8: stacking the eye region video images amplified by the eye movement frequency obtained in the step (4) according to a time sequence, and performing fast Fourier transform to obtain an eye region video image frequency spectrum;

and step 9: extracting a frequency corresponding to a frequency spectrum peak value of the eye region video image frequency spectrum obtained in the step 8 as a monitoring result of the eye movement frequency;

step 10: building a sleeping posture monitoring neural network structure based on deep learning, executing a step 11 if the sleeping posture monitoring neural network is not trained, and executing a step 13 if the sleeping posture monitoring neural network is trained;

step 11: collecting more than 1000 images of a tester in sleep in advance through three cameras arranged on the human body, the left camera and the right camera, carrying out feature marking on the obtained images, manually marking the sleeping posture state of the tester, and correspondingly sleeping in six sleeping postures, namely supine, prone, left-side straight lying, left-side crouching, right-side straight lying and right-side crouching;

step 12: sending the image data marked in the step 11 into a neural network for training, dividing a training set and a verification set according to the proportion of 8:2 to train the network until the accuracy of the verification set reaches more than 95%, and finishing the network training;

step 13: inputting the three-position body video images of the testee in the three cameras in the step 1 into a trained neural network, and carrying out six classification outputs on the images, wherein the classification results correspond to six sleeping positions of supine, prone, left-side lying straight, left-side lying crouch, right-side lying straight and right-side lying crouch;

step 14: if the tester switches any two sleeping postures of the six sleeping postures in the step 13, the tester records the switching as one-time turning, but the mutual conversion of the left-side lying and the mutual conversion of the right-side lying and the right-side lying do not take the statistics of the turning times;

step 15: and comprehensively evaluating the sleep quality result of the tester by combining the heart rate parameters obtained in the step 7, the eye movement frequency obtained in the step 9, the sleeping posture state obtained in the step 13 and the turn-over times obtained in the step 14.

Wherein, the step 2 specifically comprises the following steps:

step 2.1: calling a dlib library in python for a video image acquired by a camera positioned above the body of a tester, segmenting an area where a human face is positioned, and extracting a video image of the face of the tester;

step 2.2: performing face key point detection on the face video image obtained in the step 2.1, calling a dlib library in python to perform face key point detection, and obtaining the positions of 68 key points of the face of the tester;

step 2.3: carrying out image segmentation on the video image obtained in the step 2.1 again through the positions of the key points of the human face obtained in the step 2.2; segmenting a forehead area video image of a rectangular area by identifying key points of the centers of left and right eyebrows of the human face and the upper boundary of the human face identified by a dlib library;

step 2.4: finding out key points representing a left eye corner, a right eye corner, the uppermost part of an eye socket and the lowermost part of the eye socket by identifying key points of left and right eyes of the human face, and segmenting an eye area video image of a rectangular area through the four key points;

wherein, the step 3 specifically comprises the following steps:

step 3.1: the encoder structure of the deep convolutional neural network for extracting and amplifying the physiological signal is constructed by the following steps: after each convolution layer of the 2 convolution layers, a Relu activation function is used, then 3 residual error networks are used, physiological signals in the sleep video are extracted through one convolution layer, the step length of the physiological signals is set to be 2, and finally two residual error structures are connected for output;

step 3.2: building a modulation amplification structure of a deep convolutional neural network, performing convolutional operation on the difference of physiological signals of two frames of sleep images through a convolutional layer, enabling an activation function to be a Relu function, multiplying the activation function by an amplification factor alpha, and performing nonlinear change on the amplified characteristics by using the convolutional layer and a residual error structure to obtain amplified physiological signal difference characteristics;

step 3.3: and building a decoder structure of a deep convolutional neural network, superposing the amplified physiological signal differential characteristics to the initial sleep image, and decoding and outputting the amplified video through upsampling and two convolutional layers to realize the amplification of the physiological signal in the sleep video.

Wherein, the step 4 specifically comprises the following steps:

step 4.1: inputting the forehead area video image obtained in the step 2 into the deep convolutional neural network built in the step 3, setting an amplification factor alpha to be 15, extracting and amplifying the heart rate of the forehead area, and outputting the forehead area video image after amplifying the heart rate signal;

step 4.2: inputting the eye region video image obtained in the step 2 into the deep convolutional neural network constructed in the step 3, setting an amplification factor alpha to be 30, extracting and amplifying the eye movement frequency of the eye region, and outputting the eye region video image after amplifying the eye movement frequency;

wherein, the step 10 specifically comprises:

step 10.1: building a neural network structure, which comprises 4 convolutional layers, 3 maximum pooling layers, 1 full-connection layer and 1 classifier;

step 10.2: preventing the calculated amount from being too large, extracting the current-time key frames of the video every 1s, respectively extracting 1 frame of images from the cameras positioned on the left and right of the body of the tester, and forming three-channel image data to be input into the neural network in the step 10.1;

step 10.3: extracting image features of the three-channel image data in the step 10.2 through a convolution layer with convolution kernels of 10 x 10, a maximum pooling layer of 2 x 2, a convolution layer with convolution kernels of 10 x 10, a maximum pooling layer of 2 x 2 and a convolution layer with convolution kernels of 10 x 10 respectively;

step 10.4: inputting the extracted features into a full-connection layer for six classifications, wherein classification results correspond to six sleeping postures: supine, prone, left supine, left crouch, right supine, and right crouch.

Wherein, the step 15 specifically comprises:

step 15.1: the heart rate of a person in normal sleep is 60-100 times per minute, the heart rate of the person in deep sleep can be reduced to 50 times per minute, the heart rate of the tester in sleep is monitored, when the heart rate is obviously reduced, the tester is considered to enter a deep sleep period, when the heart rate is gradually increased, the tester is considered to exit the deep sleep period, finally, the proportion of the deep sleep period in sleep of the tester is counted, and the higher the proportion of the deep sleep period is, the higher the sleep quality is;

step 15.2: when the eye movement is in a rapid eye movement period in a sleep stage, the eyeballs can rotate rapidly, the eye movement of the tester is monitored, if the eye movement frequency is obviously increased, the heart rate is increased in the step 14.1, the tester can be considered to enter the rapid eye movement period, the duration time of the rapid eye movement period is counted, and if the rapid eye movement sleep is suddenly interrupted, the signal of the attack of diseases such as angina pectoris and asthma is often generated;

step 15.3: the sleeping posture is considered to be a better sleeping posture during sleeping, but the sleeping posture is not suitable for people suffering from respiratory diseases or people who frequently snore, and the sleeping posture of lying on the side is adopted; monitoring the sleeping posture of the tester, and if the tester has respiratory diseases or snores, carrying out sleeping posture adjustment suggestion when the tester adopts a non-side sleeping posture;

step 15.4: the turnover frequency of the tester is monitored, and if the turnover frequency is too high, the tester is prompted to possibly lack calcium ions or have high mental stress and poor sleep quality.

The invention relates to a non-contact sleep monitoring system based on deep learning and multi-parameter fusion, which comprises the steps of firstly segmenting an acquired video image, segmenting a forehead area video image, an eye area video image and a three-direction body video image, then building a deep convolutional neural network for extracting and amplifying physiological signals, amplifying a heart rate signal of a forehead area and an eye movement frequency of an eye area by setting different amplification factors of the network to obtain a forehead area video image for amplifying the heart rate signal and an eye area video image after amplifying the eye movement frequency, extracting corresponding frequency spectrums by utilizing fast Fourier transform, and finding out frequencies corresponding to frequency spectrum peaks as the monitored heart rate signal and eye movement frequency. For three-direction body video images, a sleeping posture monitoring neural network structure based on deep learning is built, sleeping posture features extracted from the network are input into a full connection layer for six classification, classification results correspond to six sleeping postures of supine, prone, left-side straight lying, left-side crouching, right-side straight lying and right-side crouching, and d number of turn-over times is counted through switching of different sleeping postures of the six sleeping postures. And finally, comprehensively evaluating the sleep quality by integrating the monitored physiological signals. The invention provides a sleep monitoring system with high comfort, multi-parameter fusion and high automation for testers, realizes the monitoring of physiological parameters such as heart rate, respiratory rate, eye movement frequency, sleeping posture, turning-over times and the like in a non-contact manner, improves the monitoring reliability, and has key effects on clinical diagnosis of sleep quality and clinical treatment and early intervention on sleep disorder patients or potential patients.

Drawings

FIG. 1 is a deep convolutional neural network diagram of physiological signal extraction and amplification

FIG. 2 is a flow chart of heart rate monitoring

FIG. 3 is a flow chart of eye movement monitoring

FIG. 4 is a flow chart for monitoring sleeping posture and turning-over times

FIG. 5 is a diagram of a sleeping posture monitoring neural network structure

Detailed Description

The following describes a non-contact sleep monitoring system based on deep learning and multi-parameter fusion in detail with reference to the accompanying drawings:

step 1: and (5) building a sleep monitoring platform. Three cameras are respectively arranged at the upper, left and right positions of the body of a tester to acquire video images of the tester in the sleeping process;

step 2.2: performing face key point detection on the face video image obtained in the step 2.1, calling a dlib library in python to perform face key point detection, and obtaining positions of 68 key points of the face of the tester;

step 2.3: and (4) carrying out image segmentation on the video image obtained in the step (2.1) again through the positions of the key points of the human face obtained in the step (2.2). Segmenting a forehead area video image of a rectangular area by identifying key points of the centers of left and right eyebrows of the human face and the upper boundary of the human face identified by a dlib library;

step 2.4: and finding out key points representing the left eye corner, the right eye corner, the uppermost part of the eye socket and the lowermost part of the eye socket by identifying the key points of the left eye and the right eye of the human face, and segmenting an eye area video image of a rectangular area by the four key points.

step 3.1: constructing an encoder structure of a deep convolutional neural network for extracting and amplifying physiological signals, wherein the encoder structure comprises 2 convolutional layers and 3 residual error networks, a Relu activation function is used after each convolutional layer, then the physiological signals in the sleep video are extracted through one convolutional layer, the step length of the physiological signals is set to be 2, and finally the two residual error structures are connected for output;

step 3.3: and building a decoder structure of a deep convolutional neural network, superposing the amplified physiological signal differential characteristics to an initial sleep image, and decoding and outputting the amplified video through upsampling and two convolutional layers to realize amplification of the physiological signals in the sleep video.

and 4.2: inputting the eye region video image obtained in the step 2 into the deep convolutional neural network constructed in the step 3, setting an amplification factor alpha to be 30, extracting and amplifying the eye movement frequency of the eye region, and outputting the eye region video image after amplifying the eye movement frequency;

step 6: performing fast Fourier transform on the pulse wave signals obtained in the step 5 to obtain a time series frequency spectrum of the human pulse wave;

and step 8: stacking the eye region video images amplified by the eye movement frequency obtained in the step (4) according to a time sequence, and performing fast Fourier transform to obtain an eye region video image frequency spectrum;

and step 9: extracting the frequency corresponding to the frequency spectrum peak value of the eye region video image frequency spectrum obtained in the step 8 as the monitoring result of the eye movement frequency;

step 10: and (3) building a sleeping posture monitoring neural network structure based on deep learning, and executing the step 11 if the sleeping posture monitoring neural network is not trained. If the sleeping posture monitoring neural network is trained, executing step 13;

step 13: inputting the three-direction body video images of the testee in the three cameras in the step 1 into a trained neural network, and carrying out six classification outputs on the images, wherein the classification results correspond to six sleeping postures of supine, prone, left-side lying straight, left-side lying crouch, right-side lying straight and right-side lying crouch;

step 14: if the tester switches any two of the six sleeping postures in the step 13, the tester records the switching as one-time turning, but the mutual conversion between the left-side lying posture and the mutual conversion between the right-side lying posture and the right-side lying posture are not counted into the statistics of the turning times;

Step 15.1: the heart rate of a person in normal sleep is 60-100 times per minute, and the heart rate can be reduced to 50 times per minute in deep sleep. The heart rate of the tester during sleeping is monitored, when the heart rate is obviously reduced, the tester is considered to enter the deep sleep period, and when the heart rate is gradually increased, the tester is considered to exit the deep sleep period. Finally, counting the proportion of deep sleep periods in the sleep of the testers, wherein the larger the proportion of the deep sleep periods is, the higher the sleep quality is;

step 15.2: in the rapid eye movement stage of the sleep stage, the eyeball rotates rapidly. The invention monitors the eye movement of the tester, if the eye movement frequency is obviously increased, the tester can be considered to enter the rapid eye movement period by combining the heart rate increase in the step 14.1, the duration time of the rapid eye movement period is counted, and if the rapid eye movement sleep is suddenly interrupted, the rapid eye movement sleep is often a signal for the attack of diseases such as angina, asthma and the like;

step 15.3: lying is considered to be a better sleeping position during sleeping, but is not suitable for people suffering from respiratory diseases or who frequently snore, and a side sleeping position is adopted. The invention monitors the sleeping posture of a tester, and if the tester has respiratory tract diseases or snores, the tester carries out sleeping posture adjustment suggestion when the tester adopts a non-side sleeping posture;

step 15.4: the invention monitors the turnover frequency of the tester, and if the turnover frequency is too high, the tester is prompted to possibly lack calcium ions or have high mental stress and poor sleep quality.

Claims

1. A non-contact sleep monitoring system based on deep learning and multi-parameter fusion comprises the following steps:

and 5: carrying out RGB three-channel separation on each frame image in the forehead area video image obtained in the step 4 after the heart rate signal is amplified, averaging pixel points in R, G, B three channels, and then carrying out time sequence stacking to obtain a pulse wave signal;

and 7: carrying out spectrum analysis on the time series frequency spectrum obtained in the step 6, and selecting the frequency corresponding to the peak value of the frequency spectrum as a heart rate monitoring result;

step 10: and (3) building a sleeping posture monitoring neural network structure based on deep learning, and executing the step 11 if the sleeping posture monitoring neural network is not trained. If the sleeping posture monitoring neural network is trained, executing the step 13;

step 15: and (3) comprehensively evaluating the sleep quality result of the tester by combining the heart rate parameters obtained in the step (7), the eye movement frequency obtained in the step (9), the sleeping posture state obtained in the step (13) and the turning times obtained in the step (14).

2. The system for monitoring sleep based on deep learning and multi-parameter fusion as claimed in claim 1, wherein the step 2 is specifically:

step 2.1: calling a dlib library in python for a video image acquired by a camera positioned above the body of the tester, segmenting a region where a face is positioned, and extracting a video image of the face of the tester;

3. The system for monitoring sleep based on deep learning and multi-parameter fusion as claimed in claim 1, wherein the step 3 is specifically:

step 3.2: building a modulation amplification structure of a deep convolutional neural network, performing convolution operation on the difference of physiological signals of two frames of sleep images through a convolutional layer, wherein an activation function is a Relu function, multiplying the activation function by an amplification factor alpha, and performing nonlinear change on the amplified characteristics by using the convolutional layer and a residual structure to obtain amplified physiological signal difference characteristics;

4. The system for monitoring sleep based on deep learning and multi-parameter fusion as claimed in claim 1, wherein the step 4 is specifically:

step 4.2: and (3) inputting the eye region video image obtained in the step (2) into the deep convolutional neural network built in the step (3), setting an amplification factor alpha to be 30, extracting and amplifying the eye movement frequency of the eye region, and outputting the eye region video image after the eye movement frequency is amplified.

5. The system according to claim 1, wherein the step 10 is specifically as follows:

6. The system according to claim 1, wherein the step 15 is specifically as follows:

step 15.1: the heart rate of a person in normal sleep is 60-100 times per minute, and the heart rate can be reduced to 50 times per minute in deep sleep. The invention monitors the heart rate of a tester during sleeping, and considers that the tester enters a deep sleep period when the heart rate is obviously reduced, and considers that the tester exits the deep sleep period when the heart rate is gradually increased. Finally, counting the proportion of deep sleep periods in the sleep of the testers, wherein the larger the proportion of the deep sleep periods is, the higher the sleep quality is;

step 15.2: during the rapid eye movement phase of the sleep cycle, the eye ball rotates rapidly. The invention monitors the eye movement of the tester, if the eye movement frequency is obviously increased, the tester can be considered to enter the rapid eye movement period by combining the heart rate increase in the step 14.1, the duration time of the rapid eye movement period is counted, and if the rapid eye movement sleep is suddenly interrupted, the rapid eye movement sleep is often a signal for the attack of diseases such as angina, asthma and the like;