WO2022121913A1

WO2022121913A1 - Contactless baby monitor

Info

Publication number: WO2022121913A1
Application number: PCT/CN2021/136239
Authority: WO
Inventors: Sze Chuen CHAN; Ming Yip Wallace WONG
Original assignee: Actywell Digital Ltd
Priority date: 2020-12-08
Filing date: 2021-12-08
Publication date: 2022-06-16
Also published as: DE202021106628U1

Abstract

A method of monitoring a sleeping baby using two cameras. The readings of cameras are used to cancel the opposing physical movements of the baby to obtain a pure reading of the baby's breathing.

Description

CONTACTLESS BABY MONITOR

FIELD OF THE INVENTION

The present invention relates to the contactless physiology sensing systems. In particular, the present invention relates to contactless baby movements and heart rate monitors.

BACKGROUND

Inadequate sleep is one of the major risk factors of Sudden Baby Death Syndrome (SIDS) . Every year there are about15,000 deaths globally, and it is the 3rd leading cause of children under the age of 1 in the USA.

Sound detectors and motion detectors have been proposed for monitoring sleeping babies. If the baby breathes steadily, very low levels of sound and motions may be detected, and the parents may listen to and watch the baby remotely via a monitor and a speaker to know that the baby is not under distress. Typically, these detectors take input from a microphone and/or a camera placed near the baby. The camera typically has night vision functions, such as by detecting in infrared. If the baby’s breathing sounds and movements indicate that the baby has awakened, the devices set off an alarm to notify its parents. However, these kinds of devices are mainly useful for monitoring the baby for general sleep quality but cannot really detect the onset of SIDS to warn the parents of such an emergency, as babies tend not to put up a detectable struggle when unable to breathe.

Another proposal to monitor SIDS is to use physiological sensors. Some of these sensors monitor the baby’s heart rate to determine if the baby is comfortable. The heart rate is able to show that the baby is under stress even if there is no physical indication from the baby showing the stress. However, all these sensors are only able to sense heart rate by means of a device attached to the baby. The device comprises a harmless, invisible infrared light that is shone onto the skin to observe surges of blood in the skin as the heart pumps. In this way, the pulse or heart rate of the baby is monitored. An alarm will be raised when heart rate anomaly is detected. However, baby skin is delicate and not suitable for continual contact with many kinds of materials. Hence, there are concerns of skin allergy and hygiene from continual contact of the baby with the device, and also concerns of device heat hurting the baby.

To avoid having sensors that require contact with the baby, the MikuTM Smart Baby Monitor proposes baby clothing with special patterns printed thereon. The patterns can be observed by a remote camera and recognised by software. Any positional changes of the patterns can be used to deduce movements of the baby’s chest, heart-rate and respiration rate. However, the consumable cost for this method is high for obvious reasons. Furthermore, the clothing needs to be regularly cleaned and changed.

Eulerian Video Magnification (EVM) has also been proposed to detect heart-rate visually, i.e. using colour magnification to allow human perception of the pulse which is otherwise imperceptible. Typically in a digital video clip, very tiny change of one colour to another colour in every pixel of every frame in a video is unnoticeable. EVM simply amplifies such changes by creating a greater colour change than the original video has recorded. Colour amplification means replacing the original, tiny colour change with an even greater change of colour further along the colour spectrum. Videos processed this way show dramatic changes in colour where the colour changes were originally more subtle. As a result, flushes of blood on the skin for every beat in the pulse are made visible in the processed video. However, this technology requires very sensitive cameras that can detect subtle changes in colour before the colour change may be amplified. It follows that better data may be obtained with greater illumination. However, a bright environment is unsuitable for babies to sleep in. Furthermore, complex calculation of colour amplification in real time requires huge processing power, which translates directly into high costs.

Accordingly, an improved method of monitoring the heart rate of sleeping babies, over those of the described prior art, is desirable.

STATEMENT OF INVENTION

In a first aspect, the invention proposes a method of monitoring a subject, comprising steps of: providing at least two cameras; placing each camera in a different angle to observe the same movement of the subject and obtaining an image the subject; such that the same movements observed by each of the at least two cameras are in different phases; wherein the same movement in different phases can be used to cancel out each other.

By observing the signal of the at least two cameras, it is possible to separate observations caused by movements and observations caused by flushing of blood. This allows vital signs of the subject to be monitored.

Light variations caused by movements includes, but are not limited to, movements induced by breathing, vibrations of the body caused by cardiac activities, voluntary and non-voluntary movements of the subject, or movements of the subject caused by reaction to regular situations or ambient conditions, such as heat of the day.

Preferably the system observes the variation of signals at different wavelengths.

Preferably, one of the at least two cameras is capable of detecting in invisible wavelengths. More preferably, one of the at least two cameras is capable of detecting in infrared.

Optionally, both the at least two cameras are capable of detecting in infrared. Yet in another option, the at least two cameras are capable of detecting in any invisible wavelengths, including different or identical wavelengths.

Optionally, the system provides at least one light source to allow the use of device in a dark environment. Preferably, one of the at least the light sources is invisible light source.

In a second aspect, the invention proposes a system for monitoring a subject, comprising: a plurality of cameras; each camera placed in a different position to observe a subject in a different angle.

BRIEF DESCRIPTION OF DRAWINGS

It will be convenient to further describe the present invention with respect to the accompanying drawings that illustrate possible arrangements of the invention, in which like integers refer to like parts. Other embodiments of the invention are possible, and consequently the particularity of the accompanying drawings is not to be understood as superseding the generality of the preceding description of the invention.

Figure 1 shows an embodiment of the invention;

Figure 2 illustrates how Eulerian Video Magnification works;

Figure 3 illustrates the light variation captured by different cameras due to different factors;

Figure 4 illustrates the light variation captured by different cameras due to different factors at different wavelengths;

Figure 5 shows a variation of the embodiment of Figure 1;

Figure 6 show a comparative example to the embodiment of Figure 1; and

Figure 7 show charts corresponding to the readings of Figure 6.

DESCRIPTION OF EMBODIMENT

Figure 1 shows an embodiment that comprises at least two

cameras

101, 103 to capture movements of a sleeping baby 105 from different angles. Each of the cameras possesses Eulerian Video Magnification (EVM) functions to detect the pulse transit time or heart rate of the baby 105.

Pulse felt in the extremities of the body, i.e. the limbs and head, do not take place exactly at the same time as the heart contracts. This is due to the distance from the heart to the limbs and head. Pulse transit time is the time required for the flush of blood as pumped by the heart to reach the extremities of the body.

Typically, pulse transit time tends to be observed by measuring the exact moment the heart contracts, using electrocardiography (ECG) , and timing how long does it takes for the surge of blood to be observable in one of the extremities such as a wrist, using photoplethysmography (PPG) sensors. The time difference between the ECG and the PPG signals is taken to be the pulse transmit time for that particular limb. However, in the present embodiment, no PPG sensor is required to be attached to a limb of a baby 105.

Eulerian Video Magnification

EVM is a signal processing method to increase temporal variations in videos that are difficult or impossible to see with the naked eye and to display them in an easily visible manner. Typically, the method takes a standard video sequence as input, and applies spatial decomposition, followed by temporal filtering to the frames. The resulting signal is then amplified to reveal subtle information visually.

As explained in an earlier section, EVM simply amplifies colour changes in the by creating an even greater colour change for each pixel that has changed colour in the original video. This means replacing the original, tiny colour change with another colour further along the colour spectrum. Videos processed this way show dramatic changes in colour where the colour changes were originally more subtle. As a result, flushes of blood on the skin for every beat in the pulse are made visible in the processed video.

Figure 2 is an illustration that shows how EVM colour amplification makes the subtle flush of blood in a baby’s face much more apparent. The drawing shows how the pulse would look in four consecutive frames of a video. The difference between each left frame and the very next right frame is in the milliseconds. The left most drawing shows the baby’s face without blood, and corresponds to the time when the heart has just contracted, at 401, before the surge of blood has reached the face.

The surges of blood caused by the pumping of the heart make up the pulse, and each surge passes over the head very quickly. Any skin colour change is not visible to the naked eye. Therefore, the flow of blood as it fills the face which is not normally noticeable to the eye becomes noticeable in the processed video using EVM technology. Even small physical movements may also be amplified to become noticeable. This technique can be used to process video images in real time.

The surge of blood 409 in the arteries due the heart contraction eventually reaches the neck in the second drawing from the left, at 403, illustrated by a different colour on the chin of the baby 105. The surge of blood 409 then advances upwardly and reaches lower half of the face of the baby 105 in the third drawing, at 405. Finally, the surge of blood 409 reaches the forehead in the fourth and rightmost drawing, at 405. Blood returning to the heart by the veins do not exhibit a surge.

Subsequently, measurement of the speed in which the surge of blood 409 passes over the face provides the pulse transit time. As the pulse transit time is a measurement of how much time does the flush of blood reaches the extremity of the body after the heart has contracted, simply measuring the speed that the flush goes over the short distance of the face of the baby is able to provide an accurate estimate of the pulse transit time. From the timing observed of each flush of blood, the heart rate of the baby is also obtained.

Observing the pulse in the dark

In prior art applications of EVM, sufficient illumination is required for a single camera to detect minute changes in skin colour, pixel by pixel. However, bright illumination is not desirable when one is trying to keep the baby asleep. The embodiment reduces this need for bright illumination by use of the two or more cameras that are able to detect in invisible wavelengths, preferably infrared. Accordingly, skin colour changes need not be captured in a fully illuminated environment, since the EVM data captured by the two or more cameras in infrared can be combined to observe the pulse transit time accurately, particularly when detecting in infrared wavelengths as the flush of warm blood can be observed easily. Accordingly, the embodiment provides the possibility of non-contact pulse transit time and heart rate measurement.

Cancelling movement from pulse measurement

Movements of the baby 105 may affect the accuracy of pulse transit time measurements made on the EVM treated video. This is because some movements of the baby 105 may be misinterpreted to be part of the surge of blood 409 in the baby’s face. In the right direction, the baby’s movement may move the flush of blood as observed by the cameras faster in the direction the blood is travelling in. This creates a false image of a faster pulse.

Movements can be eliminated, however, by using the two cameras, so that it becomes possible to observe the baby 105’s pulse transmit time or heart rate more accurately.

Figure 3 shows in greater detail how the embodiment of Figure 1 is able to separate the pulse from movements using at least two cameras. The two cameras are placed on opposite sides of the baby, so that each camera views the baby from a different side, i.e. one from the left and one from the right of the baby.

The pulse signals obtained by the left camera 101, i.e. by observing the flush of blood across the baby’s face made observable using EVM technology, is shown in the top left chart 501. The pulse signals obtained by the right camera 103 is shown in the left bottom chart 503.

The movement signals observed by the left camera 101 is shown in the top right chart 511. The movement signals observed by the right camera 101 are shown in the right bottom chart 513.

As shown in the charts in Figure 3, 501, 503, 511, 513, pulse signals of the baby taken by any number of cameras are always in phase. Hence, all the signals can be added together to amplify the pulse.

However, when the baby moves away from one of the cameras 101, the baby has to be moving towards the other camera 103 at the same time. In this way, use of two

cameras

101, 103 capture the same movement as signals in opposite phases. In other words, the same movement captured by the two different cameras produces movement signals with opposite amplitudes. Depending on the positions of the cameras, the amplitudes of the movement signals may be the same. If so, the movement signals can be completely cancelled out by adding the output of the two cameras together.

In practice, however, the baby does not move away from one camera and move towards the other camera, all along the principal axes of the cameras. The baby may move diagonally away from one camera, and move towards the other camera in a completely different angle. Hence, the amplitude of the movement signals may be weighted or adjusted so that as much of the movement signals are eliminated as possible.

The pulse signal can never be obscured as any summation would only strengthen the pulse signal. Hence, in any case, the pulse signal can only be improved.

In other words, while the movement signals from the different cameras may not be exactly opposite to each other, they are highly inversely or negatively correlated, while the pulse signals are always positively correlated. Hence, using simple mathematical treatments, such as principal components analysis (PCA) , or deploying deep learning machine, it is possible to eliminate or separate the movement signal from the pulse signal.

One advantage of using multiple cameras is that this allows cameras of lower product specification to be used while achieving a high level of monitoring of the baby 105’s movements.

As mentioned, the

cameras

101, 103 record the baby 105 in invisible wavelengths. In one variation of the embodiment, one of the cameras records the baby 105 in ultraviolet wavelength while the other one of the cameras records the baby 105 in an infrared wavelength. Alternatively, more than one of the cameras record the baby 105 in ultraviolet wavelength, or more than one of the cameras record the baby 105 in infrared wavelengths.

In another embodiment, the pulse signal is observed at the two palms of the baby, which allows the pulse transit time to each palm to be deduced. In another embodiment, the time difference between the pulse as observed in a palm and a foot may be used to deduced the pulse transit time.

In another embodiment, the time delay between movement signal and pulse signal is used to observe the pulse transit time, which cannot be achieved without separating the movement and pulse signal.

In another embodiment, both

cameras

101, 105 are capable of capturing light of different wavelengths. These observations are made when each of the two cameras observe the baby in different wavelengths. By separating the movement signals from the pulse signals, it is possible to compare the pulse signals measured in different wavelengths and obtain other vital signs. For example, it is possible to use one of the cameras to observe a wavelength that is emitted by oxygenated blood, while use the other one of the cameras to observe a wavelength that is emitted by any blood. In this case, the ratio of oxygenated blood versus overall blood can be observed, leading to a detection of oxygen saturation, or SpO2 for short. Accordingly, Figure 4 shows two pulse, the top pulse 601 being for oxygenated blood having an amplitude p while the bottom pulse 603 for all blood having amplitude q. p/q will provide the SpO2.

Typically, the wavelengths for monitoring SpO2 are such that one of the wavelengths in in the visible red wavelength, and the other is in the infrared.

Figure 5 shows an even more preferable embodiment in which three cameras are used to observe the baby 105. Preferably, the three

cameras

101, 103, 302 are mounted on a single triangular frame (not illustrated) , with one camera on each corner. This ensures that the cameras are capable of triangulating any point on the baby 105. The more cameras are used, the easier it will be for the body movements to be eliminated to obtain the pulse.

Without cancellation of the movements hiding the pulse signals, accurate oxygen saturation cannot be measured using cameras placed at a remote distance from the baby.

Research has demonstrated that there is a possibility of detecting blood pressure by observing time difference in the reflection of light of different wavelengths from the same location (https: //pubmed. ncbi. nlm. nih. gov/30307851/) . Hence, by separating the movement signal with pulse signal, it is possible to compare the time different of reflective of light with different wavelengths.

In another embodiment, face-detection is also used to determine the location of the face, and thereby identify where the chest is. This is because the chest is the more obvious regularly moving part as the baby 105 sleeps. Hence, once the face is detected, the each camera automatically observes regular chest movements below the face. Typically, the distance is about 10 to 20 cm below the face, although the embodiment needs to calculate the angle that represents the 10 to 20cm based on the distance the embodiment is placed from the baby 105.

Preferably, one or more light-source is added to the embodiments to eliminate the need of ambient light. Preferable the light source gives infra-red light and so is invisible to human eyes and is harmless to human.

The embodiment may also be used in area of patient monitoring and elderly monitoring, or people in coma, instead of babies.

Comparative example

A comparative example given in Figure 6 and Figure 7 that shows why a single camera 201 is unable to separate the two signals. Figure 6 shows a baby monitored by a single monitor, while Figure 7 shows charts representing the baby’s pulse 211 and bodily movements 221, respectively. The single camera has EVM in Figure 6 functions to observe the pulse of the baby. The vertical motions of the baby are represented by the two-headed arrow along the y-axis in Figure 6. Thereby, top chart 211 in Figure 7 shows the pulse of the baby as may be observed by EVM technology. Physical, external movements of the baby, which is always present affects and distorts the readings of the pulse, as shown in both the

charts

211, 221. The top chart 211 in Figure 7 shows the pulse as it should be observed by EVM. The bottom chart 221 shows signals of the movements, such as chest heavings. When the pulse signals and movement signals overlap or coincide, there is no way to separate the pulse from the movements. In the prior art, this problem is not addressed, as the subjects being monitored are simply assumed be stationary.

While there has been described in the foregoing description preferred embodiments of the present invention, it will be understood by those skilled in the technology concerned that many variations or modifications in details of design, construction or operation may be made without departing from the scope of the present invention as claimed.

For example, the described embodiments may include an alarm to detect abnormalities in the movements and breathing of a baby under monitoring.

Furthermore, both

cameras

101, 103 may observe the baby 105 in the same infrared or ultraviolet wavelength. Alternatively, however, one camera may observe the baby 105 in a ultraviolet wavelength while the other camera may observe the baby 105 in infrared wavelength.

In a variation of the embodiment of Figure 1, embodiments in which one or both of the

cameras

101, 103 are placed at the top of the baby 105 is within contemplation of this embodiment, as long as the cameras are each at an angle to the baby so different that movements can be cancelled.

Claims

A method of monitoring a subject, comprising steps of:

providing at least two cameras;

placing each camera in a different angle to obtain a video recording of the subject;

processing the video obtained by each camera to observe the pulse of the subject;

adding videos obtained by each camera together to eliminate the same movements captured by the cameras in the different angles;

such that

movement artefacts in the observation of the pulse of the subject is eliminated.
A method of monitoring a subject as claimed in claim 1, comprising steps of:

wherein one of the at least two cameras is capable of obtaining the video recording of the subject in an invisible wavelength.
A method of monitoring a subject as claimed in claim 2, comprising steps of:

The invisible wavelength is an infrared wavelength.
A method of monitoring a subject as claimed in claim 3, comprising steps of:

wherein both the at least two cameras is capable of detecting in infrared.
A system for monitoring a subject, comprising:

a plurality of cameras;

each camera placed in a different position to observe a subject in a different angle.
A system for monitoring a subject as claimed in claim 5, wherein:

at least one of the plurality of cameras is configured to observe flush of blood in the skin of the subject.
A system for monitoring a subject as claimed in claim 6, further comprising:

a calculation module for cancelling the effects of movements on the observation of the flush of blood in the skin of the subject.
A system of monitoring physiological parameters, as claimed in claim 7 wherein

the flush of blood is observed in an invisible wavelength.
A system for monitoring a subject as claimed in claim 5, further comprising:

detecting the face of a subject; such that

the chest of the subject is identified by estimating the chest to be below the face; and

monitoring the chest for movements.