WO2020157755A1

WO2020157755A1 - Smart interactive sleep monitor

Info

Publication number: WO2020157755A1
Application number: PCT/IL2020/050110
Authority: WO
Inventors: Omri ARGAMAN; Yaniv Shtalryd
Original assignee: Hisense Ltd
Priority date: 2019-01-29
Filing date: 2020-01-29
Publication date: 2020-08-06

Abstract

Methods and systems for monitoring and attending to someone sleeping include: receiving one or more sensor signals, including a bed motion signal, indicative of a sleeper's presence and of one or more vital signs; processing the sensor signals to identify one or more signal characteristics that correspond to one or more sleeping conditions, including at least one of a sleep status, a sleep cycle, a room condition preference, and an at-risk condition; and responsively to identifying the one or more signal characteristics, providing an instruction to one or more room devices to affect a room condition, according to a predefined correspondence between the one or more signal characteristics and one or more automated actions.

Description

SMART INTERACTIVE SLEEP MONITOR

FIELD OF THE INVENTION

[0001] The invention generally relates to accessories for sleep monitoring and especially for supporting care by caregivers.

BACKGROUND

[0002] Caregivers (such as parents) who are responsible for the well-being of their wards, such as an infant, an elderly person, or someone who is infirm, may occasionally have to step away from the person in their care, especially when the person is sleeping. Nevertheless, the caregiver needs to remain aware in any changes in their ward's state, especially when the person is waking up. People requiring assistance, for example when leaving their bed, may also be unable to articulate their needs and optimal comfort levels. Providing support for caregivers (such as parents) in addressing needs of their ward and in increasing the ward's comfort and security, without constant direct attention, could improve the overall level of care.

SUMMARY

[0003] Embodiments of the present invention provide methods and systems for monitoring and attending to needs of a caregiver's ward when the ward is sleeping or resting in a bed. An embodiment of the present invention provides a computer-implemented method of monitoring and attending to someone sleeping, including steps of: receiving one or more sensor signals, including a bed motion signal, indicative of a sleeper's presence and of one or more vital signs; processing the sensor signals to identify one or more signal characteristics that correspond to one or more sleeping conditions, including at least one of a sleep status, a sleep cycle, a room condition preference, and an at-risk condition; and responsively to identifying the one or more signal characteristics, providing an instruction to one or more room devices to affect a room condition, according to a predefined correspondence between the one or more signal characteristics and one or more automated actions.

[0004] The one or more room devices may include one or more of a room light, a room audio speaker, and a crib vibrator. The instruction to one or more room devices to affect a room condition may be at least one of operating a vibrating mechanism, playing an audio recording, setting a room light intensity, and setting a room light color. The sensor signal may be from one or more of a bed motion sensor, an audio sensor, and a video monitor. The vital signs may include one or more of a heartbeat rate, a heartbeat rate variability measure, and a breathing rate. The sleep status may be one of awake or asleep. The at-risk condition may be an indication of sudden infant death syndrome (SIDS). The predefined correspondence between the one or more signal characteristics and the one or more automated actions may be determined as a correspondence to achieve one or more of helping a person to sleep, extending a person's sleep time, adjusting room conditions for a caregiver to provide care, and alerting a caregiver to an at-risk condition.

[0005] Some embodiments further include providing a status notification to a caregiver indicative of the corresponding one or more sleeping conditions. The status notification may be provided at regular time intervals. The status notification may be an alert transmitted immediately upon determining the presentation of the one or more sleeping conditions. The status notification may be transmitted to a mobile device of a caregiver.

[0006] Some embodiments further include providing a status report to a caregiver including one or more details with respect to: an infant's sleep pattern and vital signs; an image captured by a video monitor; a times that the infant was attended, an identity of an attending caregiver; and a time at which an infant motion of a given type was identified. [0007] Some embodiments further include receiving a room condition signal indicative of temperature or humidity, comparing the room signal condition to a set of predefined levels, and responsively changing, automatically, one or more of a room temperature setting, an air purifier setting, and a humidity setting.

BRIEF DESCRIPTION OF DRAWINGS

[0008] For a better understanding of various embodiments of the invention and to show how the same may be carried into effect, reference will now be made, by way of example, to the accompanying drawings. Structural details of the invention are shown to provide a fundamental understanding of the invention, the description, taken with the drawings, making apparent to those skilled in the art how the several forms of the invention may be embodied in practice. In the accompanying drawings:

[0009] Fig. 1 is a schematic block diagram of a system for monitoring and attending to the needs of someone sleeping, according to some embodiments of the present invention; and

[0010] Fig. 2 is a flow diagram of a method for monitoring and attending to needs of someone sleeping, according to some embodiments of the present invention.

DETAILED DESCRIPTION

[0011] It is to be understood that the invention and its application are not limited to the methods and systems described below or to the arrangement of the components set forth or illustrated in the drawings, but are applicable to other embodiments that may be practiced or carried out in various ways.

[0012] Embodiments of the present invention provide methods and systems for monitoring and supporting the well-being of someone sleeping.

[0013] Fig. 1 is a schematic block diagram of a "smart bedroom" system 20, according to some embodiments of the present invention. The smart bedroom system (also referred to hereinbelow as the "smart bedroom") includes smart bedroom devices for sensing room conditions in a room 22 in which the smart bedroom operates, the room conditions including sounds, lighting, air parameters (quality, humidity, temperature, etc.), and the vital signs of a person sleeping, i.e., a person being monitored. The bedroom may be an infant's nursery (in which case the bed is typically an infant's crib).

[0014] The smart bedroom also includes one or more smart bedroom devices for affecting the room conditions so as to affect a sleeper's sleep pattern, thereby improving the sleeper's well-being, including the sleeper's comfort and safety.

[0015] In some embodiments, sensor signals from smart bedroom sensors are transmitted to a main processor, which processes the signals to determine appropriate, real-time settings for the operation of the one or more smart bedroom devices. The main processor may be a cloud-based, remote server 24, which may communicate with the smart bedroom devices through a smart bedroom hub 26. Communications between the smart bedroom devices (including sensors) and the smart bedroom hub 26 may be by any wired or wireless communications protocol, such as the Bluetooth Low Energy (BLE) protocol. In further embodiments, the smart bedroom hub, rather than being an independent device may be integrated into one of the other smart bedroom devices described hereinbelow.

[0016] The remote server may be configured by a machine learning training process to optimize the operation of room devices in response to the received sensor signals, that is, to optimize the control signals that are sent to the room devices as described further hereinbelow with respect to Fig. 2. The optimization of control signals is typically designed to provide optimal sleeping conditions, which may include conditions for putting a person to sleep, extending a person's sleep, and avoiding health risks such as sudden infant death syndrome

(SIDS). [0017] In some embodiments of the present invention, a human operator, such as a caregiver, may operate a mobile device 28 on which an application, referred to hereinbelow as a "smart bedroom app," may be installed. The smart bedroom app may provide a human interface for interacting with the smart bedroom, including interacting with the remote server, the hub, the sensors and the devices for performing tasks, such as setup, monitoring, and maintenance. The mobile device 28 may communicate by wireless means, such as Bluetooth or Wi-Fi Direct wireless standards with smart bedroom sensors and devices, either directly, or through the smart bedroom hub 26. In further embodiments, the mobile device 28 may communicate by mobile device communication protocols with the cloud-based, remote server 24, which in turn provides a communications path, typically by internet communications protocols, to the smart bedroom hub 26. The mobile device may be a smart phone, such as an Android compatible device. Other processing devices— such as work stations, smart TVs, and internet of things (IOT) devices— may also may also be configured to execute a similarly configured smart bedroom app for providing a user interface, thereby enabling a caregiver to interact with the other smart bedroom devices.

[0018] Typical smart bedroom sensors for sensing room conditions include a microphone 30, a video monitor 32, a bed motion sensor 34 (which may operate as an“under- the-mattress” pressure sensor) and an air quality sensor 36 (which may include multiple sensors for sensing aspects of the air, such as particulates, humidity, and temperature).

[0019] The bed motion sensor 34 is typically an“under-the-mattress” sensor pad that is configured to capture a sleeper's vital signs, such as heart rate and respiratory rate. In some embodiments, rather than sending pressure or motion signals from the bed motion sensor directly to the remote server, the bed motion sensor may process the signals and transmit a post-processed signal at regular intervals indicating a sleeper's vital signs including heart rate, heart rate variability (HRV) and/or breathing rate. Additional processing functions, described further hereinbelow, may include recognizing, based on the vital signs and movement, a person's sleep status (whether sleeping or not, level of relaxation, etc.) as well as learning infant sleeping patterns. Processing by the remote server may include determining appropriate settings of the smart bedroom devices based on the infant's sleep status and sleeping patterns.

[0020] The bed motion sensor may also monitor whether or not a person is in the bed (or a baby is in its crib). When there is no one in bed, other monitor and control functions may be automatically stopped. The bed motion sensor may also be configured to automatically transmit an immediate alarm signal if an infant has stopped breathing, indicating that the infant is experiencing SIDS.

[0021] The video monitor (i.e., camera) and/or the microphone may be configured to monitor a sleeper's presence in the bed. If a person is not present, some or all automated actions of the smart bedroom, such as issuing controls and alerts, are typically disabled. The video monitor (i.e., camera) and microphone may also be configured to sense a person's vital signs, determining a person's breathing, for example, from audio and/or visual signs. Alternatively or additionally, the video monitor and microphone may monitor other respective video and audio signals in the room, such as movement, light levels, human features, such as facial features (for facial recognition) and voices (for voice recognition). The video monitor may also include infrared sensitivity for detecting low light images. As mentioned above, in some embodiments the smart bedroom hub is integrated into one of the sensors or devices, such as the video monitor.

[0022] Signals from the various sensors are typically transmitted to the smart bedroom hub 26, which in turn may transmit the signals to the remote server 24 for processing that determines settings of the smart bedroom devices. Alternatively or additionally, some or all of the processing may be done directly by the smart bedroom hub or by the mobile device 28.

The smart bedroom devices that affect various conditions in the sleeper's environment may include a bed motion generator 38, a speaker 40, an air purifier 42, a humidifier 44, a thermostat 46, and a light 46.

[0023] The motion generator 38 is typically configured to apply soothing motion, such as a rocking motion, to the bed, and, in particular, to an infant's crib. Typically the type of motion can be controlled in real time, such that the motion may be applied at different frequencies (i.e., resonances) and at different power levels. Such rocking vibrations may sooth people, particularly infants.

[0024] The speaker in some embodiments may be a "white noise" generator, programmed to generate low frequency sounds like a "rumble," which may help to calm a someone trying to fall asleep. Typically, the speaker is also configured to play additional sounds, such as a soft heartbeat, which may mimic the sound of a mother's heart, or soft music. The various audio options may be pre-recorded and stored in the speaker, or in some embodiments the audio may be transmitted from the mobile device or remote server.

[0025] The air purifier 42, the humidifier 44, and the thermostat 46 may all be set remotely from the mobile device or remote server. Settings may be modified according to signals received related to room conditions and a sleeper's vital signs, as described further hereinbelow.

[0026] The light 48 is typically a red-green-blue (RGB) variable night light, which can be modified in real time to change light levels and colors.

[0027] Fig. 2 is a flow diagram of a process 200 for monitoring and attending to sleeper's needs, according to some embodiments of the present invention. The process includes an operational stage 202 that typically operates continuously, regardless of whether or not a person is in the bed. As described above, recognition of a person's presence may be one of the functions of the system, such that stage 202 may operate regardless of whether a person is present or not in the bed of the smart bedroom. [0028] At a step 204, signals from the smart bedroom sensors are acquired for processing, typically, by the remote server 24. (Alternatively or additionally, processing may be performed at the smart bedroom hub 26 and/or at the mobile device 28.)

[0029] At a step 206, the acquired signals are processed by signal processing methods to extract characteristics, or features, of those signals that correspond to predefined rules. The predefined rules indicate automated actions to be implemented when given characteristics of the signals are identified. Examples of such actions are described further hereinbelow. In some embodiments, the predefined rules may be configured as a neural network or as other frameworks of artificial intelligence known in the art. Alternatively or additionally, the predefined rules may be configured as a rules table, specifying signal thresholds that activate the corresponding automated actions. The rules may include recognition methods for identifying caregivers, based on audio and/or visual signals. The rules may also include recognizing sleep patterns, including expected remaining sleep time, based on motion signals received from the bed motion sensor. Automated actions defined by the rules may include setting environmental controls to optimize comfort for the sleeper, including helping the sleeper to fall asleep and extending sleep time.

[0030] At a step 208, the application of the rules determined at the step 206 may include issuing control signals to operate and/or change settings of smart bedroom devices, such as the light, speaker, bed vibrator, etc. Additional actions may include transmitting signals, including notices and alerts, to the mobile device 28, as well as to other possible user interfaces. Actions performed at the step 208 are typically performed in parallel with the continued acquisition of sensor signals and the processing of those signals. That is, the steps of the operational stage 202 are generally performed continuously as a real-time monitoring and control process. [0031] The predefined rules may be generated and/or improved by a training, or machine learning, stage 210, which may operate in parallel with the operational stage 202. At a configuration step 220, operational goals are set, such as goals of helping a person to fall asleep more quickly, extending sleep time, or setting sleep goals according to the time of day. At a validation step 222, the goals are compared with actual results of the current smart bedroom device settings. The validation step determines whether current rules are meeting the goals, and, if not, changes are made to the rules according to preset change patterns. Change patterns may include, for example, incremental lighting and audio changes.

[0032] Modifications to the rules may be applied at a step 224. In some embodiments, stage 210 is a machine learning framework, such as a neural network generator. The machine learning may be based on multiple iterative tests of one smart bedroom, or on multiple tests from multiple smart bedrooms. In some embodiments, the training process "learns" correlations between vital signs, sleeper motions, and room conditions (e.g., lighting, crib vibration, and sounds), in order to determine the sleeper's "preferences," that is, conditions such as sound, lighting, and crib vibration patterns, that may help the person to fall asleep or extend the person's sleep time. The correlations with the sleepers preferences are then incorporated into the preset rules. Stage 210 is typically implemented by the remote server 24; but alternatively or additionally, this processing may be performed at the smart bedroom hub 26 and/or at the mobile device 28.

[0033] Subsequently, during the operational stage, the main processor may provide to a caregiver a notification of expected remaining sleep time (or expected time until falling asleep). A caregiver may have an option for modifying the room conditions to affect the sleeper's sleep patterns, such as extending the sleep time (for example, by changing the lighting, bed vibration, and sounds). [0034] The various scenarios described below indicate typical signal processing situations in which the smart bedroom sensor signals are received by the main processor and the main processor responsively changes device settings.

[0035] In one exemplary scenario, the main processor monitors movement, video, and/or audio signals and determines an sleeper's sleep patterns (also referred to herein as "sleep cycles"), including making a determination of when a person is about to wake up. In response, if the caregiver has set the smart bedroom to extend sleep time to a maximum, the main processor may start playing sounds determined to be soothing (such as white noise), to help the sleeper to continue sleeping.

[0036] In another exemplary scenario, when a caregiver enters the smart bedroom room and approaches his sleeping ward at night, the main processor turns on the smart bedroom light to a preset low light level, to help the caregiver care for the person. The light is then turned off when received signals, such as video monitor signals, indicate that the caregiver is exiting the room.

[0037] In another exemplary scenario, particularly designed for infants, the main processor detects from audio signals that a mother has attached the microphone to her chest. The heartbeats are then played on the room’s speaker, soothing an infant in real-time.

[0038] From the mobile device, the caregiver can also select a given item of music, such as a lullaby, from the smart bedroom app, or other audio item, such as a parent reading a story. The app transmits the selection to the main processor, which then plays the selected item from the smart bedroom speakers. The preset rules may also determine lighting and crib motions that are appropriate for the given item. Audio items, such as lullabies, may also be recorded by the caregiver from the smart bedroom app, and stored so that they can be subsequently selected for playing. [0039] In another exemplary scenario, the main processor may determine a person's sleeping behavior and sleeping status, and may notify the caregiver that the current sleep is expected to continue for a certain period of time.

[0040] In another exemplary scenario, the main processor detects that a sleeping person is not moving or in distress, and responsively notifies the caregiver with critical information about the nature of the emergency. During this time, the main processor may also instruct the in-room speakers to play alarm sounds, as caregivers may be in hearing range.

[0041] In another exemplary scenario, based on signals from one or more of the video monitor, the microphone, and the bed motion sensor, the main processor detects that there is no person in the bed. Rules for controlling the environment for encouraging sleep may not be activated until someone returns to bed (or an infant is set in the crib).

[0042] In another exemplary scenario, when the air sensor detects that humidity is below a desired rate, the humidifier is instructed by the main processor to start working. This action may be set to operate continuously or only when a person is in the bed.

[0043] In another exemplary scenario, comprehensive reports about an person's sleeping cycle are provided to the caregiver (that is, to the smart bedroom app on the mobile device). The report may include sleep analytics, vital signs, images captured by the video monitor, the times that the sleeper was attended, and by whom (which may be determined according to image or voice processing), the number of movements and times the sleeper moved, or turned over, etc.

[0044] Computational aspects of system 20 and of process 200 may be implemented in digital electronic circuitry, or in computer hardware, embedded firmware, software, or in combinations thereof. All or part of the process may be implemented as a computer program product, tangibly embodied in an information carrier, such as a machine -readable storage device or in a propagated signal, for execution by, or to control the operation of, data processing apparatus, such as a programmable processor, computer, or deployed to be executed on multiple computers at one site, or distributed across multiple sites, including cloud configurations. Memory storage may also include multiple distributed memory units, including one or more types of storage media. A computing system configured to implement the system may have one or more processors and one or more network interface modules. Processors may be configured as a multi -processing or distributed processing system. Network interface modules may control the sending and receiving of data packets over networks.

[0045] It is to be understood that the scope of the present invention includes variations and modifications thereof which would occur to persons skilled in the art upon reading the foregoing description and which are not disclosed in the prior art.

Claims

1. A method of monitoring and attending to someone sleeping, implemented by a controller having one or more processors and having memory storage, the memory storage including instructions that when executed by the one or more processors implement steps comprising: receiving one or more sensor signals, including a bed motion signal, indicative of a sleeper's presence and of one or more vital signs;

processing the sensor signals to identify one or more signal characteristics that correspond to one or more sleeping conditions, including at least one of a sleep status, a sleep cycle, a room condition preference, and an at-risk condition; and

responsively to identifying the one or more signal characteristics, providing an instruction to one or more room devices to affect a room condition, according to a predefined correspondence between the one or more signal characteristics and one or more automated actions.

2. The method of claim 1, wherein the one or more room devices include one or more of a room light, a room audio speaker, and a crib vibrator.

3. The method of claim 1, wherein the instruction to one or more room devices to affect a room condition is an instruction to control least one of operating a vibrating mechanism, playing an audio recording, setting a room light intensity, and setting a room light color.

4. The method of claim 1, wherein the sensor signal is a signal from one or more of a bed motion sensor, an audio sensor, and a video monitor.

5. The method of claim 1, wherein the vital signs include one or more of a heartbeat rate, a heartbeat rate variability measure, and a breathing rate.

6. The method of claim 1, wherein the sleep status is one of awake or asleep.

IB

7. The method of claim 1, wherein the at-risk condition is an indication of sudden infant death syndrome (SIDS).

8. The method of claim 1, wherein the predefined correspondence between the one or more signal characteristics and the one or more automated actions is determined as a correspondence to achieve one or more of helping a person to sleep, extending a person's sleep time, adjusting room conditions for a caregiver to provide care, and alerting a caregiver to an at-risk condition.

9. The method of claim 1, further comprising providing a status notification to a caregiver indicative of the corresponding one or more sleeping conditions.

10. The method of claim 9, wherein the status notification is provided at regular time intervals.

11. The method of claim 9, wherein the status notification is an alert transmitted immediately upon determining the presentation of the one or more sleeping conditions .

12. The method of claim 9, wherein the status notification is transmitted to a mobile device of a caregiver.

13. The method of claim 1, further comprising providing a status report to a caregiver including one or more details with respect to: an infant's sleep pattern and vital signs; an image captured by a video monitor; a times that the infant was attended, an identity of an attending caregiver; and a time at which an infant motion of a given type was identified.

14. The method of claim 1, further comprising receiving a room condition signal indicative of temperature or humidity, comparing the room signal condition to a set of predefined levels, and responsively changing, automatically, one or more of a room temperature setting, an air purifier setting, and a humidity setting.