KR102435808B1 - Healthcare apparatus for measuring stress score - Google Patents

Abstract

A healthcare device according to the present invention comprises: a BCG sensor for sensing a BCG signal of a subject; a camera for photographing the face of the subject to obtain a color image of the face; and a processor for detecting a region of interest (ROI) corresponding to a face from the color image of the face, detecting a first color image for a forehead portion from the detected ROI and converting the same into a black-and-white image to obtain a first black-and-white image, detecting a second color image for a cheek portion and converting the same into a black-and-white image to obtain a second black-and-white image, applying the obtained first black-and-white image and the obtained second black-and-white image to a predetermined pre-trained multitask learning algorithm model to output a remote photoplethysmography, (rPPG) signal waveform of the subject, calculating a first heart rate variability from the sensed BCG signal waveform to calculate a first stress index based on the first heart rate variability, calculating a second heart rate variability from the output remote PPG signal waveform to calculate a second stress index based on the second heart rate variability, and outputting the stress index of the subject based on the first stress index and the second stress index. Therefore, information on the heart rate, the respiratory rate, the stress index and the sleeping state of a subject can be measured accurately in a non-contact manner.

Description

HEALTHCARE APPARATUS FOR MEASURING STRESS SCORE

The present invention relates to a healthcare device, and more particularly, to a healthcare device for predicting a subject's heart rate, respiration rate, sleep state, etc. in a non-contact manner.

There are studies on human diseases by sensing biosignals with PPG sensors in a non-invasive way. However, with the current COVID-19 pandemic, technology that can remotely monitor health in a non-invasive way has become very important. Countries are recommending the use of telehealth strategies whenever possible to reduce the risk of COVID-19 in healthcare settings. Therefore, a new method is required rather than the existing method of monitoring biometric information through a sensor that requires contact with the body.

Remote health monitoring technology is based on communication systems such as cell phones and online health portals. These remote health monitoring technologies may be required very closely to continuous patient monitoring even after a pandemic like COVID-19 is over. It is used to measure the user's physiological signals based on the facial video stream using the camera. These technologies can be used not only for infectious diseases, but also for monitoring the biometric information of infants and the elderly or the monitoring of mental health.

However, as a remote health monitoring technology, there is no research or health care product that estimates a person's bio-signals and predicts heart rate, stress index, respiration rate, etc.

Korean Registration Publication No. 10-1712002

A technical problem to be achieved in the present invention is to provide a healthcare device.

Another technical problem to be achieved in the present invention is to provide a monitoring method for a healthcare device for healthcare.

Another technical problem to be achieved in the present invention is to provide a computer-readable recording medium in which a program for executing a monitoring method for healthcare in a computer is recorded.

The technical problems to be achieved in the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the description below. will be able

In order to achieve the above technical problem, a healthcare device according to an embodiment of the present invention includes a BCG sensor for sensing a ballistocardiogram (BCG) signal of a subject; a camera for photographing the subject's face to obtain a face color image; and detecting a region of interest (ROI) corresponding to a face from the face color image, detecting a first color image for a forehead among the detected regions of interest, and converting it into a black and white image to obtain a first black and white image After acquiring and detecting a second color image for the cheek region, a second black-and-white image is obtained by converting it into a black-and-white image. By applying the learning algorithm model to output a remote PPG (remote photoplethysmography, rPPG) signal waveform of the subject, and calculating a first heart rate variability from the sensed BCG signal waveform, a first stress index is calculated based on the first heart rate variability calculating, calculating a second heart rate variability from the output remote PPG signal waveform to calculate a second stress index based on the second heart rate variability, and calculating a second stress index based on the first stress index and the second stress index It may include a processor that outputs a stress index of . The output stress index may be an average value of the first stress index and the second stress index. The predetermined multi-task learning algorithm model may use Siamese Neural Networks (SNN). The healthcare device may further include a bed on which the subject can lie, and the BCG sensor may be provided to be attached to the inner surface of the cover covering the bed. The processor applies the acquired first black-and-white image and the second black-and-white image to a predetermined multi-task learning algorithm model to output a respiration rate signal waveform of the subject, and based on the output respiration rate signal waveform to calculate the respiration rate, based on the calculated respiration rate, it is possible to determine whether the subject's sleep apnea. The healthcare device may further include a communication unit for transmitting the calculated respiration rate to the interlocked terminal.

The processor detects two eye region images in the region of interest, detects two pupil images from the detected two eye region images, and detects and recognizes both irises from the detected two pupil images, so that the subject is awake (Wake) state can be determined. The processor may determine that the subject is in a sleep state after a state in which both iris are not recognized in the detected two pupil images for a predetermined time.

The healthcare device may further include a bed on which infants and toddlers corresponding to the subject can lie down, and the processor may control to maintain a bounce function in operation on the bed when it is determined that the subject is in an awake state. The healthcare device may include a bed on which the subject can lie, and the processor may control vertical and horizontal movements of the bed to gradually stop the bounce function when it is determined that the subject is in a sleep state. The healthcare device may further include a body frame, and the camera may move on the body frame so that the subject's face is photographed in consideration of the subject's lying posture and direction.

The healthcare device may further include a communication unit for transmitting the output heart rate to an interlocked terminal. The healthcare device may further include a display unit for displaying the output heart rate.

In order to achieve the above technical problem, a healthcare device according to another embodiment of the present invention includes: a BCG sensor for sensing a ballistocardiogram (BCG) signal of a subject; a camera for obtaining an infrared (IR) image by photographing the subject's face when the night time is set at a predetermined time or the surroundings are less than a predetermined brightness; and detecting a region of interest (ROI) corresponding to the face in the infrared image, and acquiring a first image for a forehead region and a second image for a cheek region among the detected regions of interest. , by applying the obtained first image and the second image to a predetermined multi-task learning algorithm model to output a remote PPG (remote photoplethysmography, rPPG) signal waveform of the subject, and from the sensed BCG signal waveform A first heart rate variability is calculated to calculate a first stress index based on the first heart rate variability, and a second heart rate variability is calculated from the outputted remote PPG signal waveform to obtain a second stress index based on the second heart rate variability. and a processor for outputting the subject's stress index based on the first stress index and the second stress index.

In order to achieve the above other technical problem, a monitoring method for healthcare according to an embodiment of the present invention includes: sensing a ballistocardiogram (BCG) signal of a subject; obtaining a face color image by photographing the subject's face; detecting a region of interest (ROI) corresponding to the face from the face color image; Among the detected regions of interest, a first color image for the forehead is detected and converted into a black-and-white image to obtain a first black-and-white image, and a second color image for the cheek is detected, followed by a black-and-white image converting to to obtain a second black-and-white image; outputting a remote photoplethysmography (rPPG) signal waveform of the subject by applying the obtained first black-and-white image and the second black-and-white image to a predetermined multi-task learning algorithm model; calculating a first heart rate variability from the sensed BCG signal waveform and calculating a first stress index based on the first heart rate variability; calculating a second heart rate variability from the outputted remote PPG signal waveform and calculating a second stress index based on the second heart rate variability; and outputting the stress index of the subject based on the first stress index and the second stress index.

The method includes: detecting two eye region images in the region of interest, and detecting two pupil images in the detected two eye region regions; and determining that the subject is in a awake state when both iris are recognized from the detected two pupil images. The method may further include determining that the subject is in a sleep state when a state in which both iris are not recognized in the detected two pupil images has passed for a predetermined time. The method includes the steps of applying the obtained first black-and-white image and the second black-and-white image to a predetermined multi-task learning algorithm model to output a respiration rate signal waveform of the subject; calculating the respiration rate based on the output respiration rate signal waveform; and determining whether the subject has sleep apnea based on the calculated respiration rate. The method may further include transmitting the calculated respiration rate to an interlocked terminal. The method may further include the step of displaying the calculated respiration rate through a display unit.

In order to achieve the above other technical problem, a monitoring method for healthcare according to another embodiment of the present invention includes: sensing a ballistocardiogram (BCG) signal of a subject; acquiring an infrared (IR) image by photographing the subject's face at a preset night time or when the surrounding is less than a predetermined brightness; detecting a region of interest (ROI) corresponding to the face in the infrared image; acquiring a first image of a forehead and a second image of a cheek among the detected regions of interest; outputting a remote photoplethysmography (rPPG) signal waveform of the subject by applying the obtained first image and the second image to a predetermined multi-task learning algorithm model;

A first heart rate variability is calculated from the sensed BCG signal waveform, a first stress index is calculated based on the first heart rate variability, and a second heart rate variability is calculated from the output remote PPG signal waveform to the second heart rate variability. calculating a second stress index based on and outputting the stress index of the subject based on the first stress index and the second stress index.

The healthcare device according to an embodiment of the present invention can measure information about a subject's heart rate, respiration rate, stress index, sleep state (sleep apnea), etc. in a non-contact manner, but can significantly improve accuracy.

The healthcare device according to an embodiment of the present invention can significantly improve the accuracy of the wake/sleep state recognition by the subject's iris recognition method.

The healthcare device according to an embodiment of the present invention enables continuous health monitoring by estimating biosignals in a way that predicts PPG and respiration rate (RR) for infants, the elderly, and the sick based on a face image.

The health care device according to an embodiment of the present invention has the effect of monitoring the health of infants, infants, patients, etc. by estimating bio-signals based on images in a non-contact manner in a situation where an infectious disease such as COVID-19 continues.

The effects obtainable in the present invention are not limited to the above-mentioned effects, and other effects not mentioned may be clearly understood by those of ordinary skill in the art from the following description. will be.

BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are included as a part of the detailed description to help the understanding of the present invention, provide embodiments of the present invention, and together with the detailed description, explain the technical spirit of the present invention.
1 is a diagram illustrating a layer structure of an artificial neural network.
2 is a diagram illustrating a deep trajectory (signal) waveform.
3 is an exemplary diagram for describing a remote PPG (rPPG).
4 is a block diagram for explaining the configuration of the healthcare device 400 according to an embodiment of the present invention.
5 is a diagram for explaining the output of a PPG signal and a respiration rate (RR) signal in the MTL algorithm learning model using the remote PPG technique.
6 is a diagram illustrating a PPG signal frequency analysis and LF score/HF score distribution for calculating a subject's stress index.
7 is a diagram illustrating a healthcare device 400 according to the present invention.
8 is a diagram illustrating a step in which the healthcare device 400 according to the present invention detects the subject's iris.
9 is a diagram illustrating in detail a method of detecting, by the healthcare device 400 according to the present invention, a pupil (iris) of a subject.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings. DETAILED DESCRIPTION The detailed description set forth below in conjunction with the appended drawings is intended to describe exemplary embodiments of the present invention and is not intended to represent the only embodiments in which the present invention may be practiced. The following detailed description includes specific details in order to provide a thorough understanding of the present invention. However, it will be apparent to one skilled in the art that the present invention may be practiced without these specific details.

In some cases, well-known structures and devices may be omitted or shown in block diagram form focusing on core functions of each structure and device in order to avoid obscuring the concept of the present invention. In addition, the same reference numerals are used to describe the same components throughout this specification.

Before describing the present invention, artificial intelligence (AI), machine learning, and deep learning will be described. The easiest way to understand the relationship between these three concepts is to imagine three concentric circles. Artificial intelligence is the biggest circle, followed by machine learning, and deep learning, which is driving the current artificial intelligence boom, is the smallest circle.

Hereinafter, we will look at deep learning in more detail.

Deep learning is a kind of artificial neural network (ANN) using the theory of human neural network. It is a set of machine learning models or algorithms that refer to deep neural networks (DNNs) that have more than one hidden layer (hereinafter referred to as the middle layer). Simply put, deep learning can be said to be an artificial neural network with a deep layer.

Deep neural networks can be seen as descendants of artificial neural networks, and they are the latest version of artificial neural networks that have achieved success in areas where numerous artificial intelligence technologies have failed in the past by going beyond existing limitations. If we look at the details of modeling artificial neural networks by mimicking biological neural networks, biological neurons are nodes in terms of processing units, and synapses are weights in connections. It was modeled as shown in Table 1.

biological neural network artificial neural network cell body node dendrite input Axon output synapse weight

1 is a diagram illustrating a layer structure of an artificial neural network. Just as human biological neurons do meaningful work by connecting not one, but many, in the case of an artificial neural network, individual neurons are connected to each other through synapses. By connecting a plurality of layers to each other, the strength of the connection between each layer can be updated with a weight. As such, it is used in the field for learning and cognition with its multi-layered structure and connection strength. Each node is connected by weighted links, and the entire model learns by repeatedly adjusting the weights. The weight expresses the importance of each node as a basic means for long-term memory. The artificial neural network trains the entire model by initializing these weights and updating and adjusting the weights with the data set to be trained. When a new input value comes in after training is completed, an appropriate output value is inferred. The learning principle of artificial neural networks can be seen as the process of forming intelligence from the generalization of experiences, and it is done in a bottom-up manner. In Figure 1, the case where there are two or more intermediate layers (that is, 5 to 10) is called a deep neural network, and the learning and inference model made through such a deep neural network can be referred to as deep learning. have.

An artificial neural network can play a certain role even if it has one middle layer (commonly referred to as a hidden layer, 'hidden layer') except for inputs and outputs, but as the complexity of the problem increases, the number of nodes or the number of layers increases. number should be increased. Among them, it is effective to take a multi-layered model by increasing the number of layers, but the range of application is limited due to the limitation that efficient learning is impossible and the amount of computation to learn the network is large.

Previous deep neural networks have usually been designed as forward neural networks, but recent studies have successfully applied deep learning structures to Recurrent Neural Networks (RNNs). As an example, there is a case of applying a deep neural network structure to the field of language modeling. In the case of Convolutional Neural Network (CNN), not only has it been well applied in the field of computer vision, but each successful application case is also well documented. More recently, convolutional neural networks have been applied to the field of acoustic modeling for Automatic Speech Recognition (ASR), and are being evaluated as being more successfully applied than existing models. Deep neural networks can be trained with standard error backpropagation algorithms. In this case, the weights may be updated through stochastic gradient descent using the following equation.

In the present invention, a health care device that predicts biometric information (sleep state such as heart rate, sleep apnea, wake/sleep state, etc.) and stress index in a non-contact state using only a face image information in a non-contact state using a predetermined learned algorithm model. I would like to suggest The health monitoring method that predicts PPG and respiration rate (RR) based on face images in a non-contact manner is suitable for subjects who need continuous health monitoring, such as infants and children and adults with diseases such as diabetes, as well as COVID-19 transmitted through contact. It can be a safe method of health monitoring in an epidemic environment. The present invention proposes a method of predicting a subject's heart rate, stress index, sleep apnea, and wake/sleep state using a remote PPG (rPPG) signal and a heart trajectory (BCG) signal. Hereinafter, as biometric information to be used in the present invention, the deep trajectory and PPG obtained through sensing of the human body will be briefly described.

Heart rate (Photoplethysmograph, PPG) sensor

A photoplethysmograph (PPG) sensor will be described as an example of the heart rate sensor. The PPG sensor is called an optical red pulse wave sensor. Photoplethysmograph (PPG) is a pulse wave measuring method that can measure heart rate activity or heart rate by measuring blood flow through blood vessels using optical properties of living tissues. A pulse wave is a pulsating waveform that is indicated when blood undulates in the heart, and can be measured through a change in blood flow, ie, a change in blood vessel volume, that occurs according to the diastolic and contractile action of the heart. The optical red pulse wave measurement method is a method of measuring pulse waves using light. The optical sensor detects and measures changes in optical properties such as reflection and absorption/transmission ratio of biological tissues that appear when the volume changes, and through this, pulse measurement is possible. . This method is widely used because it enables non-invasive pulse measurement and has advantages such as miniaturization and ease of use, and can be used as a biosignal sensor in a wearable device.

Ballistocardiogram (BCG) sensor

2 is a diagram illustrating a deep trajectory (signal) waveform.

The moment the blood discharged from the ventricles passes through the aorta during the cardiac cycle, a recoil is transmitted to our body. A signal obtained by measuring vibration (trajectory) according to changes in blood flow in the heart and blood vessels related to this is called a ballistocardiogram (BCG). Cardiac trajectory refers to a signal that measures trajectory according to changes in blood flow in the heart and blood vessels according to contraction and relaxation of the heart, and is an index indicating the activity of the heart, similar to an electrocardiogram.

Cardiac trajectory, similar to an electrocardiogram, is an indicator of the heart's activity state and is known to include information on cardiac output, reflux and abnormal blood flow according to damage to myocardial function. Therefore, this biosignal has the potential to be used clinically, such as cardiac function evaluation, heart disease (myocardial disorder, etc.) diagnosis, treatment effect confirmation, and recovery level observation. The deep trajectory signal can be measured through an acceleration sensor, load cell sensor, PVDF film sensor, EMFi sensor, etc. Since there is no need to attach electrodes to the body using these sensors, signals can be measured in an unconstrained/unaware state, and can be usefully used for long-term health monitoring or during daily life.

As shown in FIG. 2 , in the trajectory signal, the heartbeat pattern is represented by H, I, J, and K peaks, and the portion recognized as the actual heartbeat is the J peak. The heart rate pattern in the heart trajectory signal appears in various forms due to noise, environment, and personal influence. Usually, the I peak shows a remarkably large difference depending on the environment, measurement conditions, and individual differences, and the sizes of the H and J peaks are not constant. Also, when the I peak is small, only one of the H or J peaks may appear as a large form.

Briefly describe the telemetry method.

3 is an exemplary diagram for describing a remote PPG (rPPG).

The Remote PPG can be used to measure heart rate and heart rate variability remotely. As shown in FIG. 3 , a video may be captured with a high-resolution camera in order to remotely measure the heart rate and the like. It can be useful for a variety of physical, health, and emotional monitoring, such as monitoring drivers, the elderly, and infants. Remote PPG has the same principle as PPG, but is a non-contact measurement. As the contrast between specular reflection and diffuse reflection, it measures the change in the reflection of red, green, and blue light that changes in the skin. Specular reflection is pure light reflection from the skin. Diffuse reflection is a reflection remaining due to absorption and scattering of skin tissue that varies with changes in blood volume. Hemoglobin reflects red light and absorbs green light. 3 exemplifies red light, green light, and blue light waveforms detected after signal processing using the remote PPG technique, and it can be seen that there is a difference in the rPPG waveforms detected for each light. The remote measurement technology for remotely monitoring biometric information is very useful in that it can predict biometric information only from facial information in a non-contact manner in environments that require non-contact monitoring such as monitoring of infants and children and infectious diseases such as COVID-19. . The face image is obtained using ambient light in the vicinity of the subject as a light source.

Multi-Task Learning (MTL)

Many attempts have been made to predict biometric information using deep neural networks and machine learning. In the implementation of mobile health systems, multi-task learning (MTL) is an important approach to perform multiple tasks with limited resources. Simultaneously extract the PPG signal and the respiratory rate signal using MTL from the facial video stream. The present invention proposes a complex value-based multi-task learning (MTL) algorithm model that simultaneously processes video streams. The two facial regions consist of complex numerical data that is processed simultaneously in a complex-valued neural network architecture. This complex process allows for more efficient and accurate extraction of PPG signals and respiratory rate signals compared to real-value single-task learning algorithms.

MTL (Multi Task Learning) is a model learning method that predicts by learning two or more multiple tasks at the same time through a shared layer. By simultaneously learning related tasks, the learned representation is shared, so that each other's tasks can help model learning with a good representation. By learning, the useful information obtained can have a good influence on other tasks, contributing to a better model. And by predicting multiple tasks at the same time, it is learned as a generalized model that is stronger against overfitting, and since it was made as a single model for having to create two existing tasks separately, it helps greatly in reducing the weight of the model, and mobile devices such as smartphones more advantageous for the

4 is a block diagram for explaining the configuration of the healthcare device 400 according to an embodiment of the present invention, and FIG. 5 is a PPG signal and a respiration rate (RR) signal in the MTL algorithm learning model using the remote PPG technique. It is a drawing for explaining the items to be output.

Referring to FIG. 4 , the healthcare device 400 according to the present invention includes a processor 410 , a camera 420 , a bed 430 , a BCG sensor 440 , a display unit 450 , a communication unit 460 , and a body. It may include a frame 470 and a bed movement control unit 480 .

The camera 420 captures the face of the subject (eg, an infant lying on the bed 430 ). The camera 420 may transmit the photographed face image of the subject to the processor 410 through the communication unit 460 or the like. The processor 410 may acquire a face image of the subject. Here, the photographed face image of the subject may be an RGB image as a color image.

Referring to FIG. 5 , the processor 410 may acquire original data 510 obtained by photographing the subject's face and detect the subject's face image 520 from within the original data 510 . The processor 410 obtains ROI frames 530 and 540 from the detected face image 520, and in each ROI frame 530 and 540, a forehead region 430 and a cheek region ( 560) can be detected. The processor 410 may predict a PPG signal and a respiratory rate signal by learning a face image using a convolutional neural network (CNN), a Siamese neural network (SNN), etc., which are deep learning networks used for image processing. In the present invention, it may be preferable that the multi-task learning algorithm learning model uses a Siamese neural network (SNN). The processor 410 finds and learns a pattern by repeatedly applying a filter (kernel) to all regions of the forehead and cheek images extracted from the face image 520 . The reason for extracting the forehead and cheeks is to learn the model by using the difference between the two parts of the body as meaningful information due to the time difference of blood coming up from the heart.

The detected image for the forehead area and the image for the cheek area are color images. For convenience of description, the image 550 for the forehead area is referred to as a first color image, and the image 560 for the cheek area is referred to as a second color image. In this case, the processor 410 converts the first color image and the second color image into a black-and-white image to obtain a first black-and-white image from the first color image and a second black-and-white image from the second color image. After converting to black and white, the processor 410 learns through an MTL (eg, a multi-task Siamese network) and predicts a PPG signal and a respiratory rate signal.

As shown in FIG. 5 , the processor 410 converts the first color image for the forehead and the second color image for the cheeks into a first black-and-white image and a second black-and-white image, respectively, to form a Siamese neural network (SNN) is input to a predetermined learned multi-task learning algorithm model using The processor 410 applies the first black-and-white image and the second black-and-white image to a predetermined multi-task learning algorithm model and may be referred to as a remote PPG signal waveform of the subject (or a PPG signal waveform output by an rPPG technique). ), the respiratory rate (RR) waveform is output. As described above, the processor 410 converts a color image into a black and white image for the forehead and cheeks, respectively, and inputs it to a predetermined learned algorithm model is applied when the predetermined daytime or the surrounding is greater than or equal to a predetermined brightness It is preferable to do The processor 410 may control the camera 420 to acquire a color image by photographing the face of the subject when the camera 420 is set at the predetermined daytime or when the surrounding is more than a predetermined brightness.

On the other hand, when a predetermined night time or the surrounding is less than a predetermined brightness, the processor 410 acquires an infrared (IR) image by photographing the subject's face by the camera 420 or a camera provided separately for photographing an infrared image. can be controlled to do so. When the predetermined night time or the surrounding is less than a predetermined brightness, the camera 420 captures the subject's face to obtain an infrared (IR) image, and the processor 410 generates an ROI corresponding to the face in the infrared image ( ROI), and a first image of a forehead (forehead) and a second image of a cheek (cheek) may be acquired among the detected ROIs. Then, the processor 410 applies the first image and the second image respectively obtained from the infrared image to a predetermined multi-task learning algorithm model to obtain a remote PPG (remote photoplethysmography, rPPG) signal waveform of the subject. can be printed out. Thereafter, the processor 410 calculates a first heart rate from the sensed BCG signal waveform and a second heart rate from the output remote PPG signal waveform to calculate the heart rate of the subject based on the first heart rate and the second heart rate can be printed out.

As such, the processor 410 may select whether to acquire the subject's face as a color image or an infrared image according to a time zone or ambient brightness. The processor 410 converts a color image into a black-and-white image according to a selection based on a time zone or ambient brightness and applies it to a predetermined learned algorithm (MTL algorithm) model, or performs predetermined learning of an image obtained from an infrared image. It can be applied by inputting into the algorithm model.

The healthcare device 400 according to the present invention may include a bed 430 on which a subject (eg, an infant) can lie down. A BCG sensor 440 may be attached to the inside of the cover of the bed 430 . Since the BCG sensor 440 is provided on the inner surface of the bed cover, it is possible to sense a ballistic signal from the back, side, etc. when the subject lies down.

The processor 410 may calculate a heart rate from the heart trajectory signal waveform obtained from the BCG sensor 440, obtain a PPG signal waveform from a predetermined learned MTL algorithm model, and calculate a heart rate from the obtained PPG signal waveform. . Here, the heart rate calculated from the heart trajectory signal waveform is referred to as a first heart rate, and the heart rate calculated from the PPG signal waveform is referred to as a second heart rate. The first heart rate may be calculated as the number of detections of the J-peak in the BCG signal waveform per unit time (eg, 1 minute), and the second heart rate may be calculated as the number of peaks of the rPPG signal waveform per unit time (eg, 1 minute). The processor 410 may output the heart rate of the subject based on the first heart rate calculated from the heart trajectory signal waveform and the second heart rate calculated from the PPG signal waveform obtained using the remote PPG technique. As an example, the processor 410 may output a heart rate obtained by averaging the first heart rate and the second heart rate.

Also, the processor 410 may calculate a heart rate variability (HRV) from the heart trajectory signal waveform. As an example, the heart rate variability may be calculated by Equation 1 below. Here, the heart rate variability calculated from the heart trajectory signal waveform is referred to as a first heart rate variability.

The processor 410 may calculate the heart rate variability from the PPG waveform calculated from the rPPG technique based on Equation 2 below. Here, the heart rate variability calculated from the PPG signal waveform is referred to as a second heart rate variability.

6 is a diagram illustrating a PPG signal frequency analysis and LF score/HF score distribution for calculating a subject's stress index.

The processor 410 requires frequency analysis of the PPG signal waveform and the frequency analysis of the BCG signal waveform to calculate the stress index. 6 illustrates a frequency analysis of a PPG signal waveform as an example. The processor 410 may calculate a first stress index (score) of the subject based on the first heart rate variability, and may calculate a second stress index of the subject based on the second heart rate variability. Here, a method of calculating the first stress index and the second stress index is as shown in Equation 3 below.

In Equation 3, the signal (LF) of the sympathetic nervous system and the signal (HF) of the parasympathetic nervous system are calculated and converted into a stress index.

It will be described with reference to Equation 3 and FIG. 6 (a). LF (sympathetic nerve) is the integral value of the power spectral density of the heart rate variability between Low Frequency 0.04~0.15Hz. HF (parasympathetic nerve) is an integral value in the power spectral density graph of heart rate variability between high frequency 0.15 and 0.40 Hz. The stress index (score) is converted into a stress score by taking the natural logarithm of the LF power and HF power and calculating the LF score and HF score according to the range. This is a score indicating how far away from area 5 you are and how far you are from area 5.

The processor 410 may calculate a first stress index (score) of the subject based on Equation 1 and Equation 3, and calculate a second stress index based on Equation 2 and Equation 3 have. The processor 410 may output a stress index of the subject based on the first stress index and the second stress index. As an example, the processor 410 may output an average value of the first and second stress indices as the subject's stress index.

The processor 410 outputs a respiration rate (RR) signal waveform in addition to the PPG signal waveform based on a predetermined multi-task learning algorithm model using a Siamese neural network (SNN). The processor 410 also performs a health monitoring function of the subject, such as determining whether the subject (eg, infant) has sleep apnea based on the output respiration rate signal waveform. The processor 410 may control the outputted respiration signal waveform, respiration rate, etc. to be displayed on the display unit 450 so that a person monitoring the subject can see it. In addition, the processor 410 controls the output heart rate, the output stress index, and the output (or calculated) respiration rate to be displayed on the display unit 450 so that the health status of the subject can be monitored from the outside.

The communication unit 460 may periodically or aperiodically transmit information such as the output heart rate, the output stress index, and the output respiration rate to the terminal or server of the user performing health monitoring of the subject through WiFi, Bluetooth, etc. have. The non-periodic case is transmitted only when at least one of the output heart rate, the output stress index, and the output respiration rate exceeds a predetermined threshold. As an example, the communication unit 460 may transmit to the companion terminal only when it is determined that the output respiration rate is in the sleep apnea state. The user may check the health status of the subject by periodically or aperiodically receiving health monitoring information of the subject through his/her interlocked terminal.

7 is a diagram illustrating a healthcare device 400 according to the present invention.

Referring to FIG. 7 , the body frame 470 of the healthcare device 400 surrounds the lower support part supporting the bed 430 from the bottom, the bed 430 to prevent the subject from falling from the bed 430. It may include a side support 475 that performs. A camera 420 may be positioned on the side support 475 to photograph the subject's face. In order to track the subject's face as the subject moves while lying on the bed 430, a moving member may be provided on the side support 475 to allow the camera 420 to move. The camera 420 moves from the side support 475 of the body frame 470 in consideration of the subject's lying posture and direction, and the subject's face may be photographed.

8 is a diagram illustrating a step in which the healthcare device 400 according to the present invention detects the subject's iris, and FIG. It is a drawing explaining the method in detail.

Referring to FIG. 8 , the camera 420 captures the subject's face. The processor 410 detects an image 810 of the subject's face region, which is a region of interest, and extracts an image 820 of the subject's two eyes from the detected face image 810 . Then, the processor 410 detects two irises from the detected two-eye region image 820 . In this case, the processor 410 determines that the subject is in a awake state when both irises are detected or recognized in the detected two-eye region image 820 . Contrary to this, the processor 410 may determine that the subject is in a sleep or sleep state when a state in which both irises are not recognized in the detected two-eye region image 820 has passed for a predetermined time.

Referring to FIG. 9 , the processor 410 may acquire an eye region image 910 by detecting an eye region region in the subject's face image. The processor 410 detects a pupil from the acquired eye region image 910 to obtain a pupil region image 920 , and then acquires a pupil image 930 after removing eyebrows and noise from the pupil region image 920 . The processor 410 may detect the iris 950 based on the pupil image 930 and display it in the eye region image 940 .

Although only one eye is illustrated in FIG. 9 for convenience of explanation, the camera 420 and the processor 410 perform photographing, image processing, and the like for both eyes of the subject.

When it is determined that the subject (eg, infant) is in an awake state, the processor 410 may control the bed movement controller 480 to maintain the bounce function in operation in the bed 430 . The bed movement control unit 480 is provided under the bed 430 and performs a bounce function of the bed 430 by controlling the horizontal and vertical movements of the upper plate of the bed 430 under the control of the processor 410 . Alternatively, the processor 410 may control the bed movement controller 480 to control the horizontal and vertical movements of the upper plate of the bed 410 to gradually stop the bounce function when it is determined that the subject is in the sleep state.

In the past, as the eye position was detected using the contrast around the eyes and the position of the eyebrows, problems such as erroneously recognizing other parts of the face as the eyes occurred. However, after the processor 410 according to the present invention as described above does not find the position of the subject's eyes, but detects the iris in the image 820 of both eyes to determine whether the iris is recognized, the subject is awake Because the cognitive sleep state is estimated, the accuracy of the subject's wake/sleep state is greatly improved.

The embodiments described above are those in which elements and features of the present invention are combined in a predetermined form. Each component or feature should be considered optional unless explicitly stated otherwise. Each component or feature may be implemented in a form that is not combined with other components or features. It is also possible to configure embodiments of the present invention by combining some elements and/or features. The order of operations described in the embodiments of the present invention may be changed. Some features or features of one embodiment may be included in another embodiment, or may be replaced with corresponding features or features of another embodiment. It is obvious that claims that are not explicitly cited in the claims can be combined to form an embodiment or included as a new claim by amendment after filing.

In the present invention, the processor 410 may be implemented by hardware or firmware, software, or a combination thereof. When implementing the embodiment of the present invention using hardware, ASICs (application specific integrated circuits) or DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices) configured to carry out the present invention , field programmable gate arrays (FPGAs), etc. may be provided in the processor 410 . It may be implemented as a computer-readable recording medium in which a program for executing the monitoring method for healthcare according to the present invention in a computer is recorded.

It is apparent to those skilled in the art that the present invention may be embodied in other specific forms without departing from the essential characteristics of the present invention. Accordingly, the above detailed description should not be construed as restrictive in all respects but as exemplary. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are included in the scope of the present invention.

Claims (20)

In the healthcare device,
a BCG sensor for sensing a ballistocardiogram (BCG) signal of the subject;
a camera for photographing the subject's face to obtain a face color image; and
detecting a region of interest (ROI) corresponding to the face from the face color image,
Among the detected regions of interest, a first color image for the forehead is detected and converted into a black-and-white image to obtain a first black-and-white image, and a second color image for the cheek is detected, followed by a black-and-white image to obtain a second black-and-white image,
outputting a remote photoplethysmography (rPPG) signal waveform of the subject by applying the acquired first black-and-white image and the second black-and-white image to a predetermined learned algorithm model;
calculating a first heart rate variability from the sensed BCG signal waveform to calculate a first stress index based on the first heart rate variability;
calculating a second heart rate variability from the outputted remote PPG signal waveform to calculate a second stress index based on the second heart rate variability;
A health care device comprising a processor for outputting the subject's stress index based on the first stress index and the second stress index. The method of claim 1,
The output stress index is an average value of the first stress index and the second stress index, a healthcare device. The method of claim 1,
The predetermined learned algorithm model uses Siamese Neural Networks (SNN) of Multi-task Learning, healthcare device. The method of claim 1,
The processor is
Detecting two eye region images in the region of interest, and detecting two pupil images from the detected two eye region images,
When both iris are detected from the detected two pupil images and both are recognized, it is determined that the subject is in a awake state. 5. The method of claim 4,
The processor is
A health care device that determines that the subject is in a sleep state after a predetermined period of time in which both iris are not recognized in the detected two pupil images. 5. The method of claim 4,
Further comprising a bed on which infants and toddlers corresponding to the subject can lie down,
The processor, if it is determined that the subject is awake, to control to maintain the bounce function in operation on the bed, healthcare device. 6. The method of claim 5,
A bed on which the subject can lie,
The processor controls the vertical and horizontal movement of the bed to gradually stop the bounce function when it is determined that the subject is in a sleep state, a healthcare device. The method of claim 1,
further comprising a body frame,
The camera is a health care device that can move on the body frame so that the subject's face is photographed in consideration of the subject's lying posture and direction. The method of claim 1,
A health care device further comprising a communication unit for transmitting the output stress index to an interlocked terminal. The method of claim 1,
Further comprising a bed on which the subject can lie,
The BCG sensor is provided to be attached to the inner surface of the cover covering the bed, healthcare device. The method of claim 1,
The processor applies the obtained first black-and-white image and the second black-and-white image to a predetermined learned algorithm model to output a respiration rate signal waveform of the subject, and determine the respiration rate based on the output respiration rate signal waveform Calculating, based on the calculated respiration rate to determine whether the subject's sleep apnea, healthcare device. 12. The method of claim 11,
A health care device further comprising a communication unit for transmitting the calculated respiration rate to an interlocked terminal. In the healthcare device,
a BCG sensor for sensing a ballistocardiogram (BCG) signal of the subject;
a camera for obtaining an infrared (IR) image by photographing the subject's face when the night time is set at a predetermined time or the surroundings are less than a predetermined brightness; and
Detecting a region of interest (ROI) corresponding to the face in the infrared image,
Obtaining a first image of the forehead (forehead) and a second image of the cheek (cheek) among the detected regions of interest,
outputting a remote photoplethysmography (rPPG) signal waveform of the subject by applying the obtained first image and the second image to a predetermined learned algorithm model;
calculating a first heart rate variability from the sensed BCG signal waveform to calculate a first stress index based on the first heart rate variability;
calculating a second heart rate variability from the outputted remote PPG signal waveform to calculate a second stress index based on the second heart rate variability;
A health care device comprising a processor for outputting the subject's stress index based on the first stress index and the second stress index. In the monitoring method for healthcare,
Sensing a ballistocardiogram (BCG) signal of the subject;
obtaining a face color image by photographing the subject's face;
detecting a region of interest (ROI) corresponding to the face from the face color image;
Among the detected regions of interest, a first color image for the forehead is detected, converted into a black-and-white image to obtain a first black-and-white image, and a second color image for the cheek is detected, followed by a black-and-white image converting to to obtain a second black-and-white image;
outputting a remote photoplethysmography (rPPG) signal waveform of the subject by applying the obtained first black-and-white image and the second black-and-white image to a predetermined learned algorithm model;
calculating a first heart rate variability from the sensed BCG signal waveform and calculating a first stress index based on the first heart rate variability;
calculating a second heart rate variability from the outputted remote PPG signal waveform and calculating a second stress index based on the second heart rate variability; and
A monitoring method for healthcare, comprising the step of outputting the subject's stress index based on the first stress index and the second stress index. 15. The method of claim 14,
detecting two-eye region images in the region of interest and detecting two pupil images in the detected two-eye region region; and
The monitoring method for healthcare, further comprising the step of determining that the subject is in a awake state when both iris are recognized in the detected two pupil images. 16. The method of claim 15,
The monitoring method for healthcare, further comprising determining that the subject is in a sleep state when a state in which both iris are not recognized from the detected two pupil images has passed for a predetermined time. 15. The method of claim 14,
outputting a respiration rate signal waveform of the subject by applying the obtained first black-and-white image and the second black-and-white image to a predetermined learned algorithm model;
calculating the respiration rate based on the output respiration rate signal waveform; and
Further comprising the step of determining whether the subject's sleep apnea based on the calculated respiration rate, monitoring method for healthcare. 18. The method of claim 17,
Monitoring method for healthcare, further comprising the step of transmitting the calculated respiration rate to an interlocked terminal. In the monitoring method for healthcare,
Sensing a ballistocardiogram (BCG) signal of the subject;
acquiring an infrared (IR) image by photographing the subject's face at a preset night time or when the surrounding is less than a predetermined brightness;
detecting a region of interest (ROI) corresponding to the face in the infrared image;
acquiring a first image of a forehead and a second image of a cheek among the detected regions of interest;
outputting a remote photoplethysmography (rPPG) signal waveform of the subject by applying the obtained first image and the second image to a predetermined learned algorithm model;
A first heart rate variability is calculated from the sensed BCG signal waveform, a first stress index is calculated based on the first heart rate variability, and a second heart rate variability is calculated from the output remote PPG signal waveform to the second heart rate variability. calculating a second stress index based on and
A monitoring method for healthcare, comprising the step of outputting the subject's stress index based on the first stress index and the second stress index. A computer-readable recording medium recording a program for executing the monitoring method for healthcare according to any one of claims 14 to 19 on a computer.

Images