KR20230148073A - Artificial intelligence based digital healthcare apparatus - Google Patents

Abstract

The present invention relates to an artificial intelligence (AI)-based digital healthcare apparatus to enable continuous health monitoring. According to the present invention, the AI-based digital healthcare apparatus comprises: a ballistocardiogram (BCG) sensor for detecting a BCG signal of a subject; a camera capturing the subject's face to acquire a color image of the face; and a processor detecting a region of interest (ROI) corresponding to the face from the color image of the face, detecting a first color image for the forehead area among the detected ROIs, converting the first color image into a black and white image to acquire a first black and white image, detecting a second color image for the cheek area, converts the second color image into a black and white image to acquire a second black and white image, applying the acquired first and second black and white images to a predetermined learned multi-task learning algorithm model to output a remote photoplethysmography (rPPG) signal waveform of the subject, calculating a first heart rate variability from the detected BCG signal waveform to calculate a first stress index on the basis of the first heart rate variability, calculating a second heart rate variability from the output remote PPG signal waveform to calculate a second stress index on the basis of the second heart rate variability, and outputting a stress index of the subject on the basis of the first stress index and the second stress index.

Description

Artificial intelligence-based digital healthcare device{ARTIFICIAL INTELLIGENCE BASED DIGITAL HEALTHCARE APPARATUS}

The present invention relates to a healthcare device, and more specifically, to a healthcare device for predicting a subject's heart rate, respiratory rate, sleep state, etc. in a non-contact manner.

There is research to identify human diseases by sensing biological signals with a PPG sensor in a non-invasive manner. However, due to the current COVID-19 pandemic, technology that can remotely monitor health in a non-invasive manner has become very important. Countries are encouraged to use telehealth strategies where possible to reduce the risk of COVID-19 in healthcare settings. Therefore, a new method is required rather than the existing method of monitoring biometric information using sensors that require contact with the body.

Remote health monitoring technologies are based on communication systems such as mobile phones or online health portals. These remote health monitoring technologies may very well be required for continuous patient monitoring even after a pandemic such as COVID-19 is over. It is used to measure the user's physiological signals based on the facial video stream using a camera. These technologies can be used not only for infectious diseases, but also for monitoring the biometric information of infants and young children, the elderly, or mental health.

However, there is still no research or healthcare products that predict heart rate, stress index, respiratory rate, etc. by estimating a person's biosignals as a remote health monitoring technology.

Korean Registration Publication No. 10-1712002

The technical problem to be achieved by the present invention is to provide a digital healthcare device.

Another technical problem to be achieved by the present invention is to provide a monitoring method for digital healthcare devices for healthcare.

Another technical problem to be achieved by the present invention is to provide a computer-readable recording medium that records a program for executing a monitoring method for healthcare on a computer.

The technical problems to be achieved in the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the description below. You will be able to.

In a digital healthcare device according to an embodiment of the present invention to achieve the above technical problem,

a processor that determines whether to acquire the subject's face as a color image or an infrared (IR) image based on a predetermined time period or whether the surrounding brightness is less than or equal to a predetermined brightness; a camera that acquires a facial image by photographing the subject's face according to the decision of the processor; And detecting a region of interest (ROI) corresponding to the face in the face image, acquiring a first image of the forehead area and a second image of the cheek area among the detected ROIs, and , It may include a processor that applies the acquired first image and second image to a predetermined learned algorithm model to output a remote photoplethysmography (rPPG) signal waveform and a respiratory rate signal waveform of the subject.

The digital healthcare device further includes a BCG sensor for sensing a ballistocardiogram (BCG) signal of the subject, and the processor detects a first heart rate from the sensed BCG signal waveform and the output remote PPG signal. It may include a processor that calculates a second heart rate from the waveform and outputs the subject's heart rate based on the first heart rate and the second heart rate. The output heart rate corresponds to an average heart rate of the first heart rate and the second heart rate. The predetermined learned algorithm model uses Siamese Neural Networks (SNN) of Multi-task Learning.

The processor detects two eye area images in the region of interest, and detects two pupil images from the two detected eye area images. If both irises are detected and recognized in the two detected pupil images, the subject is awake. It can be judged to be in (Wake) state. The processor may determine that the subject is in a sleep state when a predetermined period of time elapses in which both irises are not recognized in the detected pupil images. It may further include a bed on which the infant or toddler corresponding to the subject can lie down, and if the processor determines that the subject is awake, the processor may control the bed to maintain the bounce function in operation.

The digital healthcare device includes a bed on which the subject can lie down, and when the processor determines that the subject is in a sleep state, the processor can control vertical and horizontal movements of the bed to gradually stop the bounce function. The digital healthcare device further includes a main body frame, and the camera can move on the main body frame to capture the subject's face in consideration of the subject's lying posture and direction. The digital healthcare device may further include a communication unit for transmitting the output heart rate to a linked terminal.

The digital healthcare device further includes a bed on which the subject can lie, and the BCG sensor is provided to be attached to the inner surface of a cover covering the bed. The processor may calculate a respiratory rate based on the output respiratory rate signal waveform, and determine whether the subject has sleep apnea based on the calculated respiratory rate. The digital healthcare device may control the calculated respiratory rate. It may further include a communication unit for transmitting the number to the linked terminal.

In order to achieve the above technical problem, a digital healthcare device according to another embodiment of the present invention monitors the subject's face based on whether the brightness of the surrounding area is below a predetermined time period or a predetermined brightness level. determining whether to acquire a color image or an infrared (IR) image; Obtaining a facial image by photographing the subject's face according to the decision of the processor; Detecting a region of interest (ROI) corresponding to a face in the face image; Obtaining a first image of the forehead area and a second image of the cheek area among the detected regions of interest; It may include applying the acquired first image and second image to a predetermined learned algorithm model to output a remote photoplethysmography (rPPG) signal waveform and a respiratory rate signal waveform of the subject.

Sensing a ballistocardiogram (BCG) signal of the subject; and

Calculating a first heart rate from the sensed BCG signal waveform and a second heart rate from the output remote PPG signal waveform, and outputting the subject's heart rate based on the first heart rate and the second heart rate. ,Monitoring method for healthcare.

The method includes detecting two eye region images in the region of interest and detecting two pupil images in the detected two eye region regions; And if both irises are recognized in the two detected pupil images, it may further include determining that the subject is awake. The method may further include determining that the subject is in a sleep state when a predetermined period of time elapses in which both irises are not recognized in the detected pupil images. The method includes calculating a respiratory rate based on the output respiratory rate signal waveform; And it may further include determining whether the subject has sleep apnea based on the calculated breathing rate.

The healthcare device according to one embodiment of the present invention measures information about the subject's heart rate, breathing rate, stress index, sleep state (sleep apnea), etc. in a non-contact manner, but can significantly improve accuracy.

The healthcare device according to an embodiment of the present invention can significantly improve the accuracy of determining wake/sleep status by recognizing the subject's iris.

The healthcare device according to an embodiment of the present invention enables continuous health monitoring by estimating biosignals by predicting PPG and respiratory rate (RR) of infants, the elderly, and the diseased based on facial images.

The healthcare device according to an embodiment of the present invention is effective in monitoring the health of infants, patients, etc. by estimating biosignals based on images in a non-contact manner in a situation where an infectious disease such as COVID-19 continues.

The effects that can be obtained from the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below. will be.

The accompanying drawings, which are included as part of the detailed description to aid understanding of the present invention, provide embodiments of the present invention and explain the technical idea of the present invention together with the detailed description.
Figure 1 is a diagram illustrating the layer structure of an artificial neural network.
Figure 2 is a diagram illustrating a cardiac ballistic (signal) waveform.
Figure 3 is an example diagram for explaining remote PPG (rPPG).
Figure 4 is a block diagram for explaining the configuration of a healthcare device 400 according to an embodiment of the present invention.
Figure 5 is a diagram illustrating the output of a PPG signal and a respiratory rate (RR) signal from an MTL algorithm learning model using the remote PPG technique.
Figure 6 is a diagram illustrating PPG signal frequency analysis and LF score/HF score distribution for calculating the subject's stress index.
Figure 7 is a diagram illustrating a healthcare device 400 according to the present invention.
Figure 8 is a diagram illustrating the step of detecting the iris of a subject by the healthcare device 400 according to the present invention.
Figure 9 is a diagram illustrating in detail how the healthcare device 400 according to the present invention detects the subject's pupils (iris).

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the attached drawings. The detailed description set forth below in conjunction with the accompanying drawings is intended to illustrate exemplary embodiments of the invention and is not intended to represent the only embodiments in which the invention may be practiced. The following detailed description includes specific details to provide a thorough understanding of the invention. However, one skilled in the art will appreciate that the present invention may be practiced without these specific details.

In some cases, in order to avoid ambiguity of the concept of the present invention, well-known structures and devices may be omitted or may be shown in block diagram form focusing on the core functions of each structure and device. In addition, the same components are described using the same reference numerals throughout this specification.

Before explaining the present invention, artificial intelligence (AI), machine learning, and deep learning will be explained. The easiest way to understand the relationship between these three concepts is to imagine three concentric circles. Artificial intelligence is the largest circle, followed by machine learning, and deep learning, which is leading the current artificial intelligence boom, is the smallest circle.

Below we look at deep learning in more detail.

Deep learning is a type of artificial neural network (ANN) that uses human neural network theory. It is composed of a layer structure and has one input layer and one output layer. It is a set of machine learning models or algorithms that refer to a deep neural network (DNN) with more than one hidden layer (hereinafter referred to as the middle layer). Simply put, Deep Learning can be said to be an artificial neural network with deep layers.

Deep neural network can be considered a descendant of artificial neural network, and is the latest version of artificial neural network, surpassing existing limitations and achieving success in areas where many artificial intelligence technologies had failed in the past. Looking at the modeling of an artificial neural network by imitating a biological neural network, in terms of processing units, biological neurons are used as nodes, and in terms of connectivity, synapses are used as weights. It was modeled as shown in Table 1.

biological neural network artificial neural network cell body node dendrites input Axon output synapse weight

Figure 1 is a diagram illustrating the layer structure of an artificial neural network. Just as human biological neurons connect not one but many, but many, meaningful tasks, in the case of an artificial neural network, individual neurons connect with each other through synapses. By connecting, multiple layers are connected to each other, and the connection strength between each layer can be updated with a weight. In this way, it is used in the field of learning and cognition due to its multi-layered structure and connection strength. Each node is connected by weighted links, and the entire model learns by repeatedly adjusting the weights. Weights are the basic means for long-term memory and express the importance of each node. The artificial neural network trains the entire model by initializing these weights and updating and adjusting the weights with the data set to be trained. After training is complete, when new input values come in, an appropriate output value is inferred. The learning principle of artificial neural networks can be viewed as a process in which intelligence is formed from the generalization of experience, and is done in a bottom-up manner. In Figure 1, when there are two or more middle layers (i.e. 5 to 10), the layers are considered to be deeper and are called a deep neural network, and the learning and inference model achieved through such a deep neural network can be referred to as deep learning. there is.

An artificial neural network can perform a certain role even if it has one middle layer (usually referred to as a 'hidden layer') excluding the input and output, but as the complexity of the problem increases, the number of nodes or the number of layers increases. The number must be increased. Among these, it is effective to use a multi-layer structure model by increasing the number of layers, but its scope of use is limited due to the inability to learn efficiently and the large amount of calculations to learn the network.

Previous deep neural networks have usually been designed as forward-feeding neural networks, but recent studies have successfully applied deep learning structures to recurrent neural networks (RNNs). For example, there is a case where a deep neural network structure is applied to the field of language modeling. In the case of Convolutional Neural Network (CNN), not only has it been well applied in the field of computer vision, but each successful application case is also well documented. More recently, convolutional neural networks have been applied to the field of acoustic modeling for Automatic Speech Recognition (ASR), and are evaluated to be more successful than existing models. Deep neural networks can be trained with the standard error backpropagation algorithm. At this time, the weights can be updated through stochastic gradient descent using the equation below.

The present invention provides a healthcare device that predicts bio-related information (heart rate, sleep states such as sleep apnea, wake/sleep states, etc.) and stress index using only facial image information in a non-contact state using a predetermined learned algorithm model. I would like to make a suggestion. The non-contact health monitoring method that predicts PPG and respiratory rate (RR) based on facial images is not only suitable for those who need continuous health monitoring, such as infants and young children and adults with diseases such as diabetes, but also is suitable for COVID-19, which is transmitted through contact. It can be a safe health monitoring method in an epidemic environment. The present invention proposes a method for predicting a subject's heart rate, stress index, sleep apnea, and wake/sleep status using remote PPG (rPPG) signals and ballistic cardiac chart (BCG) signals. Below, we will first briefly describe cardiac ballistics and PPG, which are acquired through sensing of the human body as biometric information to be used in the present invention.

Heart rate (Photoplethysmograph, PPG) sensor

As an example of a heart rate sensor, a PPG (Photoplethysmograph) sensor will be described. The PPG sensor is called an optical red pulse sensor. Photoplethysmograph (PPG) is a pulse wave measurement method that can determine heart rate activity or heart rate by measuring blood flow in blood vessels using the optical properties of biological tissue. A pulse wave is a pulsating waveform that appears as blood waves in the heart, and can be measured through changes in blood flow that occur due to the relaxation and contraction of the heart, that is, changes in the volume of blood vessels. The optical red pulse wave measurement method is a method of measuring pulse waves using light. Changes in optical characteristics such as reflection and absorption/transmission ratio of biological tissue that occur when volume changes are detected and measured by an optical sensor, making it possible to measure pulse. . This method is widely used because it enables non-invasive pulse measurement and has advantages such as miniaturization and ease of use, and can be used as a biosignal detection sensor in wearable devices.

Ballistocardiogram (BCG) sensor

Figure 2 is a diagram illustrating a cardiac ballistic (signal) waveform.

During the cardiac cycle, the moment the blood discharged from the ventricle passes through the aorta, it transmits a recoil to our body. The signal that measures vibration (ballistics) due to changes in blood flow in the heart and blood vessels related to this is called ballistocardiogram (BCG). Cardiac ballistics refers to a signal measuring the ballistics of changes in blood flow in the heart and blood vessels as the heart contracts and relaxes, and is an indicator of the heart's activity status similar to an electrocardiogram.

Similar to the electrocardiogram, the cardiac ballogram is an indicator of the heart's activity status and is known to include information on cardiac output, regurgitation and abnormal blood flow due to damage to myocardial function. Therefore, this biosignal has the potential to be used clinically, such as evaluating cardiac function, diagnosing heart disease (myocardial disorders, etc.), confirming treatment effects, and observing the degree of recovery. Cardiac ballistic signals can be measured through acceleration sensors, load cell sensors, PVDF film sensors, EMFi sensors, etc. Using these sensors, there is no need to attach electrodes to the body, so signals can be measured in an unconstrained/unconscious state, and can be useful for health monitoring for long periods of time or during daily life.

As shown in Figure 2, in the ballistic signal, the heart rate pattern is expressed as H, I, J, and K peaks, and the part recognized as the actual heart rate is the J peak. In ballistic signals, the heart rate pattern appears in various forms due to noise, environment, and personal influences. Usually, the I peak shows a noticeable difference depending on the environment, measurement conditions, and individual differences, and the size of the H and J peaks is not constant. Additionally, when the I peak is small, only one of the H or J peaks may appear in a large form.

Briefly explain the telemetry method.

Figure 3 is an example diagram for explaining remote PPG (rPPG).

You can measure heart rate and heart rate variability remotely using Remote PPG. As shown in Figure 3, in order to remotely measure heart rate, etc., a video can be captured with a high-resolution camera. It can be useful for a variety of physical, health, and emotional monitoring, including monitoring of drivers, the elderly, and infants. Remote PPG has the same principle as PPG, but is a non-contact measurement method. It measures changes in the reflection of red, green, and blue light on the skin as a contrast between specular reflection and diffuse reflection. Specular reflection is pure reflection of light from the skin. Diffuse reflection is a reflection that remains due to absorption and scattering by skin tissue, which varies with changes in blood volume. The principle is that hemoglobin reflects red light and absorbs green light. Figure 3 illustrates the waveforms of red light, green light, and blue light detected after signal processing using the remote PPG technique, and it can be seen that the rPPG waveforms detected for each light are also different. Remote measurement technology for remotely monitoring biometric information is highly useful in environments that require non-contact monitoring, such as infant monitoring or infectious diseases such as COVID-19, in that biometric information can be predicted using only facial information in a non-contact manner. . Face images are acquired using ambient light around the subject as a light source.

Multi-Task Learning (MTL)

Many attempts are being made to predict biometric information using deep neural networks and machine learning. In the implementation of mobile medical systems, multi-task learning (MTL) is an important approach to perform multiple tasks with limited resources. Simultaneously extract PPG signal and respiratory rate signal from facial video stream using MTL. The present invention proposes a complex value-based multi-task learning (MTL) algorithm model that processes video streams simultaneously. The two facial regions consist of complex number data that is processed simultaneously in a complex-valued neural network architecture. This complex process allows PPG signals and respiratory rate signals to be extracted more efficiently and accurately compared to real-value single-task learning algorithms.

MTL (Multi Task Learning) is a model learning method that simultaneously learns two or more tasks through a shared layer and makes predictions. By learning related tasks at the same time, learned representations are shared, so each task can help model learning with good representation. The useful information obtained through learning can have a positive impact on other tasks and contribute to becoming a better model. And by predicting multiple tasks at the same time, it is learned as a generalized model that is more resistant to overfitting. Since it was created as a single model instead of the existing two tasks that had to be created separately, it is greatly helpful in lightening the model and can be used in mobile devices such as smartphones. It is more advantageous to apply to .

Figure 4 is a block diagram for explaining the configuration of the healthcare device 400 according to an embodiment of the present invention, and Figure 5 shows the PPG signal and respiratory rate (RR) signal in the MTL algorithm learning model using the remote PPG technique. This is a drawing to explain what is being printed.

Referring to FIG. 4, the healthcare device 400 according to the present invention includes a processor 410, a camera 420, a bed 430, a BCG sensor 440, a display unit 450, a communication unit 460, and a main body. It may include a frame 470 and a bed movement control unit 480.

The camera 420 photographs the face of a subject (eg, an infant or toddler lying on the bed 430). The camera 420 may transmit the captured face image of the subject to the processor 410 through the communication unit 460, etc. The processor 410 may acquire a face image of the subject. The face image of the subject captured here is a color image and may be an RGB image.

Referring to FIG. 5 , the processor 410 may obtain original data 510 of a subject's face and detect the subject's face image 520 within the original data 510 . The processor 410 acquires region-of-interest frames 530 and 540 from the detected face image 520, and selects a forehead region 430 and a cheek region from each region-of-interest frame 530 and 540. 560) can be detected. The processor 410 can predict the PPG signal and respiratory rate signal by learning the face image using deep learning networks such as CNN (Convolutional Neural Network) and Siamese neural network (SNN) used in image processing. In the present invention, it may be desirable to use Siamese neural network (SNN) as the multi-task learning algorithm learning model. The processor 410 repeatedly applies a filter (kernel) to all areas of the forehead and cheek images extracted from the face image 520 to find and learn patterns. The reason for extracting the forehead and cheeks is to learn a model by using the difference between the two parts of the body as meaningful information due to the time difference in blood flowing from the heart.

The detected image for the forehead area and the image for the cheek area are color images. For convenience of explanation, the image 550 for the forehead area is referred to as a first color image, and the image 560 for the cheek area is referred to as a second color image. At this time, the processor 410 converts the first color image and the second color image into a black-and-white image to obtain a first black-and-white image from the first color image and a second black-and-white image from the second color image. After converting to black and white, the processor 410 learns through MTL (eg, Multi-task Siamese network) and predicts the PPG signal and the respiratory rate signal.

As shown in Figure 5, the processor 410 converts the first color image for the forehead area and the second color image for the cheek area into a first black and white image and a second black and white image, respectively, and uses a Siamese neural network (SNN). It is input into a predetermined learned Multi-task Learning algorithm model using . The processor 410 applies the first black-and-white image and the second black-and-white image to a predetermined learned Multi-task Learning algorithm model and applies the subject's remote PPG signal waveform (or may also be referred to as the PPG signal waveform output by the rPPG technique). ), outputs the respiratory rate (RR) waveform. As described above, the processor 410 converts color images of the forehead area and cheek area into black and white images and inputs them into a predetermined learned algorithm model when it is a predetermined daytime or when the surrounding brightness is higher than a predetermined brightness. It is desirable to do so. The processor 410 may control the camera 420 to obtain a color image by photographing the subject's face when it is the predetermined daytime or when the surrounding brightness is higher than a predetermined brightness.

On the other hand, when it is a predetermined night time or the surrounding brightness is less than a predetermined brightness, the processor 410 acquires an infrared (IR) image by using the camera 420 or a separately equipped camera for infrared image capturing to capture the subject's face. You can control it to do so. When it is a predetermined night time or the surrounding brightness is below a predetermined level, the camera 420 acquires an infrared (IR) image by photographing the subject's face, and the processor 410 selects an area of interest (region of interest) corresponding to the face in the infrared image. ROI) may be detected, and among the detected regions of interest, a first image of the forehead area may be acquired and a second image of the cheek area may be acquired. And, the processor 410 applies the first image and the second image, respectively acquired from the infrared image, to a predetermined learned Multi-task Learning algorithm model to generate the remote PPG (remote photoplethysmography, rPPG) signal waveform of the subject. Can be printed. Thereafter, the processor 410 calculates a first heart rate from the sensed BCG signal waveform and a second heart rate from the output remote PPG signal waveform, and calculates the subject's heart rate based on the first heart rate and the second heart rate. Can be printed.

In this way, the processor 410 can select whether to acquire the subject's face as a color image or an infrared image depending on the time of day or ambient brightness. The processor 410 converts a color image into a black and white image according to a selection based on time or ambient brightness and inputs and applies it to a predetermined learned algorithm (MTL algorithm) model, or applies an image acquired from an infrared image to a predetermined learned algorithm. It can be applied by inputting it into the algorithm model.

The healthcare device 400 according to the present invention may include a bed 430 on which a subject (eg, an infant) can lie. A BCG sensor 440 may be attached to the inside of the cover of the bed 430. Since the BCG sensor 440 is provided on the inner surface of the bed cover, it can sense ballistic signals from the back, side, etc. of the subject when he or she is lying down.

The processor 410 may calculate the heart rate from the ballistic signal waveform obtained from the BCG sensor 440, obtain the PPG signal waveform from a predetermined learned MTL algorithm model, and calculate the heart rate from the obtained PPG signal waveform. . Here, the heart rate calculated from the ballistic signal waveform is called the first heart rate, and the heart rate calculated from the PPG signal waveform is called the second heart rate. The first heart rate may be calculated as the number of detections of the J-peak in the BCG signal waveform per unit time (e.g., 1 minute), and the second heart rate may be calculated as the number of peaks in the rPPG signal waveform per unit time (e.g., 1 minute). The processor 410 may output the subject's heart rate based on the first heart rate calculated from the ballistic signal waveform and the second heart rate calculated from the PPG signal waveform obtained using the remote PPG technique. As an example, the processor 410 may output a heart rate that is the average of the first heart rate and the second heart rate.

Additionally, the processor 410 may calculate heart rate variability (HRV) from the ballistic signal waveform. As an example, heart rate variability can be calculated using Equation 1 below. Here, the heart rate variability calculated from the ballistic signal waveform is called the first heart rate variability.

The processor 410 can calculate heart rate variability from the PPG waveform calculated from the rPPG technique based on Equation 2 below. Here, the heart rate variability calculated from the PPG signal waveform is called the second heart rate variability.

Figure 6 is a diagram illustrating PPG signal frequency analysis and LF score/HF score distribution for calculating the subject's stress index.

The processor 410 requires frequency analysis of the PPG signal waveform and frequency analysis of the BCG signal waveform to calculate the stress index. FIG. 6 shows frequency analysis of the PPG signal waveform as an example. The processor 410 may calculate the subject's first stress index (score) based on the first heart rate variability and calculate the subject's second stress index based on the second heart rate variability. Here, the method for calculating the first stress index and the second stress index is as shown in Equation 3 below.

In Equation 3, the signal of the sympathetic nervous system (LF) and the signal of the parasympathetic nervous system (HF) are calculated and converted into a stress index.

This will be described with reference to Equation 3 above and (a) of FIG. 6. LF (sympathetic nervous system) is the integral value of the power spectral density of heart rate variability between Low Frequency 0.04 and 0.15 Hz. HF (parasympathetic nervous system) is the integral value in the power spectral density graph of heart rate variability between High Frequency 0.15 and 0.40 Hz. The stress index (score) is converted into a stress score by taking the natural logarithm of LF power and HF power and then calculating the LF score and HF score according to the range. Area 5 is normal, and this score indicates how far it is from area 5.

The processor 410 can calculate the subject's first stress index (score) based on Equation 1 and Equation 3, and calculate a second stress index based on Equation 2 and Equation 3. there is. The processor 410 may output the subject's stress index based on the first stress index and the second stress index. As an example, the processor 410 may output the average of the first and second stress indices as the subject's stress index.

The processor 410 outputs a respiratory rate (RR) signal waveform in addition to the PPG signal waveform based on a predetermined learned Multi-task Learning algorithm model using a Siamese neural network (SNN). The processor 410 also performs a health monitoring function of the subject, such as determining whether the subject (eg, an infant) has sleep apnea based on the output respiratory rate signal waveform. The processor 410 can control the output respiratory rate signal waveform, respiratory rate, etc. to be displayed on the display unit 450 so that the person monitoring the subject can see it. In addition, the processor 410 controls the output heart rate, output stress index, and output (or calculated) respiratory rate to be displayed on the display unit 450, thereby enabling the subject's health status to be monitored from the outside.

The communication unit 460 can periodically or aperiodically transmit information such as the output heart rate, output stress index, and output breathing rate to the user's terminal or server that monitors the subject's health through WiFi, Bluetooth, etc. there is. In the non-periodic case, transmission is performed only when at least one of the output heart rate, output stress index, and output breathing rate exceeds a predetermined threshold. As an example, the communication unit 460 may transmit the output respiratory rate to the linked terminal only when it is determined that the respiratory rate is in a sleep apnea state. The user can check the subject's health status by receiving the subject's health monitoring information periodically or non-periodically through the user's linked terminal.

Figure 7 is a diagram illustrating a healthcare device 400 according to the present invention.

Referring to FIG. 7, the main body frame 470 of the healthcare device 400 is a lower support portion that supports the bed 430 from the bottom, and surrounds the bed 430 to serve as a guard to prevent the subject from falling from the bed 430. It may include a side support 475 that performs. The camera 420 is located on the side support 475 to capture the subject's face. A moving member that allows the camera 420 to move may be provided on the side support 475 to track the subject's face as the subject moves while lying down on the bed 430. The camera 420 moves on the side support portion 475 of the main body frame 470 in consideration of the subject's lying posture and direction and can capture the subject's face.

Figure 8 is a diagram illustrating the steps in which the healthcare device 400 according to the present invention detects the subject's iris, and Figure 9 illustrates the step in which the healthcare device 400 according to the present invention detects the subject's pupils (iris). This is a drawing explaining the method in detail.

Referring to FIG. 8, the camera 420 photographs the subject's face. The processor 410 detects an image 810 of the subject's face area, which is a region of interest, and extracts an image 820 of the subject's eyes from the detected face image 810. Then, the processor 410 detects two irises from the two detected eye area images 820. At this time, the processor 410 determines that the subject is awake if both irises are detected or recognized in the two detected eye area images 820. In contrast, the processor 410 may determine that the subject is in a sleeping or sleep state when both irises are not recognized in the detected eye area image 820 for a predetermined period of time.

Referring to FIG. 9 , the processor 410 may acquire an eye region image 910 by detecting the eye region in the subject's face image. The processor 410 acquires the pupil image 920 by detecting the pupil in the acquired eye image 910, removes the eyebrows and noise from the pupil image 920, and obtains the pupil image 930. The processor 410 may detect the iris 950 based on the eye pupil image 930 and display it in the eye region image 940.

In FIG. 9 , only one eye is shown for convenience of explanation, but the camera 420 and processor 410 perform filming and image processing on both eyes of the subject.

If the processor 410 determines that the subject (eg, an infant) is awake, the processor 410 may control the bed movement control unit 480 to maintain the bounce function operating in the bed 430. The bed movement control unit 480 is provided at the bottom of the bed 430 and performs a bounce function of the bed 430 by controlling the horizontal and vertical movements of the upper plate of the bed 430 under the control of the processor 410. In contrast, if the processor 410 determines that the subject is in a sleep state, the processor 410 may control the bed movement control unit 480 to control the horizontal and vertical movements of the upper plate of the bed 410 to gradually stop the bounce function.

Previously, as the eye position was detected using the light and dark around the eyes and the position of the eyebrows, problems such as misrecognizing other parts of the face as eyes occurred. However, the processor 410 according to the present invention as described above does not find the position of the subject's eyes, but detects the iris in the two eye area images 820 and determines whether the iris is recognized, and then the subject is awake. Because it estimates whether the subject is in a cognitive sleep state, the accuracy of identifying the subject's wake/sleep state has been greatly improved.

The embodiments described above are those in which the components and features of the present invention are combined in a predetermined form. Each component or feature should be considered optional unless explicitly stated otherwise. Each component or feature may be implemented in a form that is not combined with other components or features. Additionally, it is also possible to configure an embodiment of the present invention by combining some components and/or features. The order of operations described in embodiments of the present invention may be changed. Some features or features of one embodiment may be included in other embodiments or may be replaced with corresponding features or features of other embodiments. It is obvious that claims that do not have an explicit reference relationship in the patent claims can be combined to form an embodiment or included as a new claim through amendment after filing.

In the present invention, the processor 410 may be implemented by hardware, firmware, software, or a combination thereof. When implementing embodiments of the present invention using hardware, application specific integrated circuits (ASICs) or digital signal processors (DSPs), digital signal processing devices (DSPDs), and programmable logic devices (PLDs) configured to perform the present invention. , FPGAs (field programmable gate arrays), etc. may be provided in the processor 410. The monitoring method for healthcare according to the present invention may be implemented as a computer-readable recording medium that records a program for executing the monitoring method for healthcare on a computer.

It is obvious to those skilled in the art that the present invention can be embodied in other specific forms without departing from the essential characteristics of the present invention. Accordingly, the above detailed description should not be construed as restrictive in all respects and should be considered illustrative. The scope of the present invention should be determined by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

Claims (20)

In digital healthcare devices,
a processor that determines whether to acquire the subject's face as a color image or an infrared (IR) image based on a predetermined time period or whether the surrounding brightness is less than or equal to a predetermined brightness;
a camera that acquires a facial image by photographing the subject's face according to the decision of the processor; and
Detecting a region of interest (ROI) corresponding to the face in the face image,
Among the detected regions of interest, a first image is acquired for the forehead area and a second image is acquired for the cheek area,
A digital healthcare device comprising a processor that applies the obtained first and second images to a predetermined learned algorithm model to output a remote photoplethysmography (rPPG) signal waveform and a respiratory rate signal waveform of the subject. . According to clause 1,
Further comprising a BCG sensor for sensing a ballistocardiogram (BCG) signal of the subject,
The processor calculates a first heart rate from the sensed BCG signal waveform and a second heart rate from the output remote PPG signal waveform, and outputs the subject's heart rate based on the first heart rate and the second heart rate. Including, digital healthcare devices. According to clause 2,
The output heart rate corresponds to an average heart rate of the first heart rate and the second heart rate. According to clause 1,
The predetermined learned algorithm model is a healthcare device that uses Siamese Neural Networks (SNN) of Multi-task Learning. According to clause 1,
The processor,
Detect two eye area images from the region of interest, and detect two pupil images from the two detected eye area images,
A digital healthcare device that determines that the subject is awake when both irises are detected and recognized in the two detected pupil images. According to clause 5,
The processor,
A digital healthcare device that determines that the subject is in a sleep state after a predetermined period of time when both irises are not recognized in the detected pupil images. According to clause 5,
It further includes a bed on which infants and toddlers who are eligible for the above can lie down,
The processor controls the bed to maintain the bounce function in operation when it is determined that the subject is awake. According to clause 6,
Includes a bed on which the subject can lie,
The processor controls the vertical and horizontal movements of the bed to gradually stop the bounce function when it is determined that the subject is in a sleep state. According to clause 1,
Further comprising a body frame,
The camera is a digital healthcare device that can move on the main body frame to capture the subject's face in consideration of the subject's lying posture and direction. According to clause 1,
A digital healthcare device further comprising a communication unit for transmitting the output heart rate to a linked terminal. According to clause 1,
Further comprising a bed on which the subject can lie,
The BCG sensor is provided to be attached to the inner surface of the cover covering the bed. According to clause 1,
The processor calculates a respiratory rate based on the output respiratory rate signal waveform, and determines whether the subject has sleep apnea based on the calculated respiratory rate. According to clause 12,
A digital healthcare device further comprising a communication unit for transmitting the calculated respiratory rate to a linked terminal. In the digital healthcare device monitoring method for healthcare,
determining whether to acquire the subject's face as a color image or an infrared (IR) image based on a predetermined time period or whether the surrounding brightness is less than or equal to a predetermined brightness;
Obtaining a facial image by photographing the subject's face according to the decision of the processor;
Detecting a region of interest (ROI) corresponding to a face in the face image;
Obtaining a first image of the forehead area and a second image of the cheek area among the detected regions of interest;
Applying the acquired first and second images to a predetermined learned algorithm model to output a remote photoplethysmography (rPPG) signal waveform and a respiratory rate signal waveform of the subject. Monitoring method. According to clause 14,
Sensing a ballistocardiogram (BCG) signal of the subject; and
Calculating a first heart rate from the sensed BCG signal waveform and a second heart rate from the output remote PPG signal waveform, and outputting the subject's heart rate based on the first heart rate and the second heart rate. ,Monitoring method for healthcare. According to clause 15,
Detecting two eye area images from the region of interest and detecting two eye image images from the detected two eye area areas; and
A monitoring method for healthcare, further comprising determining that the subject is awake if both irises are recognized in the detected pupil images. According to clause 15,
A monitoring method for healthcare, further comprising determining that the subject is in a sleep state when a predetermined period of time elapses when both irises are not recognized in the detected pupil images. According to clause 14,
calculating a respiratory rate based on the output respiratory rate signal waveform; and
A monitoring method for healthcare, further comprising determining whether the subject has sleep apnea based on the calculated respiratory rate. According to clause 18,
A monitoring method for healthcare, further comprising transmitting the calculated respiratory rate to a linked terminal. A computer-readable recording medium recording a program for executing the monitoring method for healthcare according to any one of claims 14 to 19 on a computer.

Images