WO2019126470A1 - Détection sans effraction de niveaux de bilirubine chez un nourrisson dans un environnement domestique intelligent - Google Patents

Détection sans effraction de niveaux de bilirubine chez un nourrisson dans un environnement domestique intelligent Download PDF

Info

Publication number
WO2019126470A1
WO2019126470A1 PCT/US2018/066753 US2018066753W WO2019126470A1 WO 2019126470 A1 WO2019126470 A1 WO 2019126470A1 US 2018066753 W US2018066753 W US 2018066753W WO 2019126470 A1 WO2019126470 A1 WO 2019126470A1
Authority
WO
WIPO (PCT)
Prior art keywords
bilirubin
infant
blood
skin
background model
Prior art date
Application number
PCT/US2018/066753
Other languages
English (en)
Inventor
Michael Dixon
Shwetak Patel
Tien Jui LEE
James Taylor
Original Assignee
Google Llc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Google Llc filed Critical Google Llc
Publication of WO2019126470A1 publication Critical patent/WO2019126470A1/fr

Links

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/0059—Measuring for diagnostic purposes; Identification of persons using light, e.g. diagnosis by transillumination, diascopy, fluorescence
    • A61B5/0077—Devices for viewing the surface of the body, e.g. camera, magnifying lens
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
    • A61B5/1032—Determining colour for diagnostic purposes
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/145—Measuring characteristics of blood in vivo, e.g. gas concentration, pH value; Measuring characteristics of body fluids or tissues, e.g. interstitial fluid, cerebral tissue
    • A61B5/1468—Measuring characteristics of blood in vivo, e.g. gas concentration, pH value; Measuring characteristics of body fluids or tissues, e.g. interstitial fluid, cerebral tissue using chemical or electrochemical methods, e.g. by polarographic means
    • A61B5/1477—Measuring characteristics of blood in vivo, e.g. gas concentration, pH value; Measuring characteristics of body fluids or tissues, e.g. interstitial fluid, cerebral tissue using chemical or electrochemical methods, e.g. by polarographic means non-invasive

Definitions

  • Newborn jaundice is a common condition in healthy newborn babies. Newborn jaundice is caused by excess bilirubin in the newborn’s blood. This bilirubin is a chemical byproduct of the processing of old blood cells. While newborn jaundice is common within healthy babies, if the bilirubin level increases to an extreme amount, irreversible brain damage or death may result. Therefore, bilirubin levels within a newborn’s blood should be monitored for a period of time after the newborn’s birth to ensure that the amount of bilirubin in the newborn’s blood does not increase to a dangerous level and is naturally decreasing.
  • the bilirubin level in a newborn’s blood tends to peak several days after birth, which is typically after a newborn has been discharged from the hospital. Therefore, it may fall to the parents of the infant to assess whether the infant’s bilirubin level warrants medical care.
  • a method of measuring bilirubin in blood may include receiving, from a streaming video camera by a bilirubin analysis system, a video feed of a portion of a room over a period of time.
  • the video feed may include skin of an infant.
  • the video feed may include a plurality of frames.
  • the method may include creating, by the bilirubin analysis system, a background model based on lighting conditions in the video feed.
  • the method may include determining, by the bilirubin analysis system, an amount of light absorption for blood present in the skin of the infant using the background model.
  • the method may include
  • the method may include outputting, by the bilirubin analysis system, the bilirubin state to an end-user device.
  • Embodiments of such a method may include one or more of the following features: Determining, by the bilirubin analysis system, that current lighting conditions match the background model.
  • the method may include, in response to determining that the current lighting conditions match the background model, determining by the bilirubin analysis system, a second amount of light absorption for blood present in the skin of the infant using the background model.
  • the method may include, in response to determining that the current lighting conditions match the background model, determining by the bilirubin analysis system, a second bilirubin state for the infant based on the amount of light absorption for the blood in the skin of the infant.
  • Outputting the bilirubin state may include outputting an indication of a change between the bilirubin state and the second bilirubin state.
  • the method may further include determining, by the bilirubin analysis system, that current lighting conditions do not match the background model.
  • the method may include, in response to determining that the current lighting conditions do not match the background model, waiting, by the bilirubin analysis system, for a period of time.
  • the method may include, in response to determining that the current lighting conditions do not match the background model, after waiting the period of time, redetermining that current lighting conditions match the background model.
  • a color calibration pattern may not be imaged as part of the video feed of the portion of the room.
  • Creating the background model may include creating a plurality of background models.
  • Each background model of the plurality of background models may be based on different lighting conditions present in the video feed.
  • the method may further include analyzing, by the bilirubin analysis system, the video feed.
  • the method may further include selecting, by the bilirubin analysis system, a background model from the plurality of background models based upon current lighting conditions present in the video feed.
  • the method may further include analyzing, by the bilirubin analysis system, the video feed.
  • the method may further include creating, by the bilirubin analysis system, an interpolated background model using two or more of the plurality of background models based upon current lighting conditions present in the video feed.
  • the interpolated background model may be used to determine the amount of light absorption for blood present in the skin of the infant.
  • the method may further include illuminating the portion of the room using a light having a stored color profile.
  • the lighting conditions within the room may be controlled by the light.
  • Illuminating the portion of the room using the light may include causing, by the bilirubin analysis system, the streaming video camera to activate an infrared light that may be integrated within a housing of the streaming video camera.
  • Illuminating the portion of the room using the light may include transmitting, by the bilirubin analysis system, an instruction to a smart home device located within the room to activate the light. The light may be incorporated as part of the smart home device.
  • a system for measuring bilirubin in blood may include a streaming video camera.
  • the streaming video camera may include a camera.
  • the streaming video camera may include a wireless network interface.
  • the streaming video camera may include one or more processors.
  • the one or more processors may cause a video feed captured using the camera to be transmitted to a cloud-based server system via the wireless network interface.
  • the video feed may include a plurality of frames that include skin of an infant and a portion of a room over a period of time.
  • a cloud-based host system, including one or more processors may be configured to receive the video feed.
  • the one or more processors may be configured to create a background model based on lighting conditions in the video feed.
  • the one or more processors may be configured to determine an amount of light absorption for blood present in the skin of the infant using the background model.
  • the one or more processors may be configured to determine a bilirubin state for the infant based on the amount of light absorption for the blood in the skin of the infant.
  • the one or more processors may be configured to output the bilirubin state to an end-user device.
  • Embodiments of such a system may include one or more of the following features: An end-user device that may execute an application.
  • the application may be configured to log into a user account linked with the streaming video camera.
  • the application may be configured to output the bilirubin state received from the cloud-based host system.
  • the one or more processors of the cloud-based host system may be further configured to determine that current lighting conditions match the background model.
  • the one or more processors of the cloud-based host system may be further configured to, in response to determining that the current lighting conditions match the background model, determine a second amount of light absorption for blood present in the skin of the infant using the background model.
  • the one or more processors of the cloud-based host system may be further configured to, in response to determining that the current lighting conditions match the background model, determine a second bilirubin state for the infant based on the amount of light absorption for the blood in the skin of the infant.
  • the one or more processors of the cloud- based host system being configured to output the bilirubin state may include the one or more processors of the cloud-based host system being configured to output an indication of a change between the bilirubin state and the second bilirubin state.
  • the one or more processors of the cloud- based host system may be further configured to determine that current lighting conditions do not match the background model.
  • the one or more processors of the cloud-based host system may be further configured to, in response to determining that the current lighting conditions do not match the background model, wait for a period of time.
  • the one or more processors of the cloud-based host system may be further configured to, in response to determining that the current lighting conditions do not match the background model, after waiting the period of time, re-determine that current lighting conditions match the background model.
  • the one or more processors of the cloud- based server system being configured to create the background model may include the one or more processors of the cloud-based server system being configured to create a plurality of background models. Each background model of the plurality of background models may be based on different lighting conditions present in the video feed.
  • the cloud-based server system may include a storage system.
  • the storage system may be configured to store the video feed mapped to a user account for at least a defined period of time.
  • the streaming video camera may further include an infrared light.
  • the one or more processors of the cloud-based server system may be further configured to cause the streaming video camera to activate the infrared light.
  • the system may further include a smart home device comprising a light.
  • the one or more processors of the cloud-based server system may be further configured to transmit an instruction to the smart home device to activate the light.
  • FIG. 1 illustrates a block diagram of an embodiment of a system for monitoring infant bilirubin levels.
  • FIG. 2 illustrates a block diagram of an embodiment of a bilirubin analysis system.
  • FIG. 3 illustrates an embodiment of a streaming video camera capturing a field-of-view including an infant and background.
  • FIG. 4 illustrates an embodiment of a smart home environment in which a bilirubin level in an infant may be monitored.
  • FIG. 5 illustrates an embodiment of a method for measuring bilirubin in blood using a fixed background model.
  • FIG. 6 illustrates an embodiment of a method for measuring bilirubin in blood using a lighting transform.
  • FIG. 7 illustrates an embodiment of a method for measuring bilirubin in blood using interpolation between multiple background models.
  • FIG. 8 illustrates an embodiment of a method for measuring bilirubin in blood using active scene illumination.
  • FIG. 9 illustrates an embodiment of an interface on an end user device for reviewing and analyzing decreasing bilirubin levels determined using a bilirubin analysis system.
  • FIG. 10 illustrates an embodiment of an interface on an end user device for reviewing and analyzing increasing bilirubin levels determined using a bilirubin analysis system.
  • Bilirubin levels within a newborn may be monitored based on properties of the newborn’s skin.
  • increased bilirubin levels in a newborn’s blood can result in the newborn’s skin having a yellowish tint. While the yellowish tint can be visibly assessed in some newborn’s skin, blue light absorption of the newborn’s skin may be a more accurate metric to use to assess an amount of bilirubin present in the blood of the newborn.
  • a camera may be used to capture an image of a newborn’s skin to assess the amount of bilirubin in the newborn’s blood; however, the lighting conditions within the region may greatly affect the measurements.
  • a color calibration pattern with predefined colors may be captured by the camera in conjunction with the infant’s skin and used to determine the lighting conditions in the vicinity of the newborn’s skin.
  • a video feed captured by a smart-home video camera may be used to assess infant bilirubin levels over time (e.g., several hours, several days, several weeks).
  • a streaming video camera may be used within an infant’s nursery, or some other location, to monitor the infant.
  • a streaming video camera may be used as an infant monitor to determine the infant’s current state (e.g., sleeping, crying, playing).
  • the streaming video camera may additionally or alternatively be used for capturing a video stream, streaming it via the Internet to a cloud-based host system, and storing the video for possible later retrieval.
  • Such video may be used for security, such as for a resident or parent to view in real time or to review at a later time to determine what occurred in the camera’s field-of-view while the resident or parent was away.
  • Frames captured as part of such a video stream may be additionally or alternatively used for determining the bilirubin level in a newborn’s blood.
  • a streaming video camera may be placed in a location such that the camera’s field-of-view at least occasionally includes the infant.
  • the camera may be aimed at a crib, bed, or playpen.
  • one or more background models may be built for at least a static portion of the field-of-view of the streaming video camera. These one or more background models may be intended to capture objects, other than the infant, that remain within the stream video camera’s field of view for lengthy periods of time and have particular visual properties.
  • These one or more background models may be developed based on the background being lit in varying lighting conditions. These background models may be used in comparing infant skin images in similar lighting conditions and/or to identify the“true” color of objects present in the background to take into account the various color properties of the one or more sources of light that are used to illuminate the background. Sources of light that vary the lighting conditions can be natural (e.g., the sun) and/or artificial (e.g., incandescent lightbulbs, fluorescent lightbulbs, LEDs, etc.).
  • a bilirubin analysis system may analyze infant skin.
  • the current lighting conditions can be taken into account when determining whether the bilirubin level within the infant is increasing, decreasing, or staying consistent.
  • FIG. 1 illustrates a block diagram of an embodiment of a system 100 for monitoring infant bilirubin levels.
  • System 100 can include: streaming video camera 110, smart home device 120, one or more networks 130, bilirubin analysis system 140, and end user device 170.
  • Streaming video camera 110 may capture a stream of video frames. These video frames may be transmitted via one or more networks to bilirubin analysis system 140.
  • Networks 130 may include a local wireless area network and the Internet.
  • a streaming video camera communicates using wired communication to a gateway device that, in turn, communicates with the Internet.
  • Streaming video camera 110 may include a wired or wireless communication interface; a camera; a microphone; one or more processors; and/or one or more integrated lights.
  • streaming video camera 110 also transmits a stream of audio to bilirubin analysis system 140.
  • streaming video camera 110 may perform functions in addition to monitoring for bilirubin levels in an infant.
  • Streaming video camera may be used as a security camera or an infant monitor.
  • Video and, possibly, audio captured by streaming video camera 110 may be stored by a cloud-based server system.
  • Such video and/or audio may also be streamed live to one or more end user devices, such as end user device 170.
  • streaming video camera 110 may transmit video directly to end user device 170.
  • the functionality of bilirubin analysis system 140 may be incorporated as part of streaming video camera 110 and/or end user device 170.
  • streaming video camera 110 may include a light that emits visible light. Such a light may be illuminated in response to a command received from a remote source, such as bilirubin analysis system 140. Additionally or alternatively, streaming video camera 110 may include one or more lights that emit other than visible light. For example, an infrared illuminator may be a type of light onboard streaming video camera 110. Again, such a light may be illuminated in response to a command received from a remote source, such as bilirubin analysis system 140.
  • Bilirubin analysis system 140 may communicate with streaming video camera 110, smart home device 120, and/or end user device 170 via one or more networks 130, which can include the Internet.
  • Bilirubin analysis system 140 may include one or more computer systems. Further detail regarding bilirubin analysis system 140 is provided in relation to FIG. 2.
  • bilirubin analysis system 140 may be incorporated as part of streaming video camera 110, smart home device 120, and/or end user device 170.
  • one or more processors incorporated as part of streaming video camera 110, smart home device 120, and/or end user device 170 can perform some or all of the tasks discussed in relation to bilirubin analysis system 140 detailed in relation to FIG. 2.
  • data may be transmitted to streaming video camera 110 for output.
  • bilirubin measurement information may be output by streaming video camera 110 via a speaker.
  • Streaming video camera 110 may monitor for movement of a person (other than the infant) or for a person interacting with the infant.
  • bilirubin information may be output, for example, as synthesized speech.
  • streaming video camera 110 and/or smart home device 120 may have integrated home assistant capabilities integrated. Such capabilities may allow a user to pose a spoken query to the device, to which the device then responds. In some embodiments, an end user may be able to use such capabilities to query either streaming video camera 110 or smart home device 120 about bilirubin levels determined by bilirubin analysis system 140.
  • Smart home device 120 can represent a smart device that is located in a same area or room as streaming video camera 110. Various forms of smart home devices are detailed in relation to FIG. 4. Smart home device 120 may have one or more integrated lights that can be activated or deactivated. In some embodiments, such one or more integrated lights may be able to be activated in a variety of colors and/or animation patterns. Bilirubin analysis system 140 may be able to activate and deactivate such an integrated light by transmitting a command to smart home device 120 via one or more networks 130. Bilirubin analysis system 140 may also be able to determine a color profile of the one or more lights of smart home device 120 by performing a database look-up or querying smart home device 120.
  • End user device 170 may be used to present results of a bilirubin analysis to an end user. Once bilirubin analysis system 140 has assessed a bilirubin level of an infant, bilirubin analysis system 140 can present the results to the end user via end user device 170.
  • End user device 170, streaming video camera 110, and smart home device 120 may each be registered with a single management account maintained by bilirubin analysis system 140. Having each device registered with a same management account may be a prerequisite to results being output to end user device 170.
  • End user device 170 can be various forms of a computerized device. End user device 170 may be a smartphone, tablet computer, desktop computer, laptop computer, smartwatch, or some other form of computerized device that can present results.
  • end user device 170 may execute a native application through which the end user is required to log into the management account linked with streaming video camera 110.
  • This application may present current and historical bilirubin measurement
  • Tips or recommendations on how the end user should behave based on the bilirubin measurements may also be presented. For example, a tip or recommendation may indicate that a call to a doctor is advisable or that bilirubin is decreasing and the infant appears healthy (at least as far as bilirubin levels are concerned).
  • FIG. 2 illustrates a block diagram of an embodiment of a cloud-based host system 200 that may host bilirubin analysis system.
  • Bilirubin analysis system 140 may be performed by cloud- based host system 200 in addition to other functions.
  • a function of cloud-based host system 200 may be to receive and store video and/or audio streams, such as from streaming video camera 110.
  • Stream processing engine 210 may receive video and, possibly, audio streams from many streaming video cameras, including video camera 110.
  • Received video and audio streams may be stored to storage 215 for at least a period of time.
  • Storage 215 can represent one or more non- transitory processor-readable mediums, such as hard drives, solid-state drives, or memory.
  • Video and audio streams 216 can represent various video (and possibly audio) streams that are stored for at least a period of time by storage 215.
  • Cloud-based host system 200 may store such video streams for a defined window of time, such as one week or one month.
  • received video may not be stored for longer than needed to perform a bilirubin analysis.
  • Management account database 217 may store account information for many user accounts.
  • Account information for a given management account can include a username, password, and/or indications of various devices linked with the management account (e.g., streaming video camera 110 and smart home device 120).
  • an end user may be able to access stored video and audio streams of video and audio streams 216 linked with that management account.
  • the end user may also be able to stream video and/or audio live (i.e., in real-time or near real-time).
  • an end user may also be able to access information relating to any bilirubin measurement performed by bilirubin analysis system 140 using a received video stream.
  • Stream processing engine 210 may perform various processing functions on received video streams. If a particular video stream is to be analyzed to determine a bilirubin level in an infant, stream processing engine 210 may route a subset or all video frames to bilirubin analysis system 140. In some embodiments, to decrease the amount of processing performed while still maintaining a high degree of accuracy in bilirubin measurements, only some frames received by stream processing engine 210 may be analyzed by bilirubin analysis system 140. For example, only one or two frames per second may be routed to bilirubin analysis system 140.
  • Bilirubin analysis system 140 may include various components that perform various processing and analysis. Such components may include: infant skin detection engine 251, background profile creator 252, infant light absorption profile creator 253, infant light absorption profile database 254, background profile database 255, bilirubin analysis engine 256, and notification engine 257. Each of these components may be implemented in the form of instructions executed using one or more processors. Such instructions may be encoded as software that is executed using one or more general-purpose processors or as firmware that is loaded to configure one or more configurable processors (e.g., FPGAs). Additionally or alternatively, one or more special-purpose processors may be configured to perform the tasks of one or more of the components.
  • Each of these components may be implemented in the form of instructions executed using one or more processors. Such instructions may be encoded as software that is executed using one or more general-
  • infant skin detection engine 251 may be used to segregate portions of a frame corresponding to skin of an infant from other portions of the frame, which are directed to background. Object and/or facial recognition may be performed to ensure the skin corresponds to an infant and/or a particular infant. Object recognition may be used to identify an infant-sized and shaped object. Movement of the infant may be used to help locate and identify the infant. The identification of an infant’s skin may be based at least in part on the infant’s skin falling within a particular range of colors.
  • Infant light absorption profile creator 253 may receive a portion of the frame that has had all background components removed. Therefore, the portion of the frame received by infant light absorption profile creator 253 may contain only an image of an infant’s skin (and, thus, the blood near the surface of the skin that affects the light reflected by the skin). Infant light absorption profile creator 253 may create an infant skin model for a portion of infant’s skin. For example, using an object recognition process, the infant’s forehead may be identified. In some
  • only a patch of skin in the center of the forehead may be used to create the infant’s skin model.
  • the measured color of the infant’s skin may be at least partially dependent on the angle of the skin to the streaming video device. As the angle of the infant’s skin to the streaming video device changes, the amount of light reflected from one or more lighting sources within the ambient environment off the infant’s skin to the streaming video device will change.
  • the infant’s skin model may be created for a portion of skin that approximately (e.g., within 5 degrees) is perpendicular to the line- of-sight of the video camera.
  • infant skin profiles may be created by infant light absorption profile creator 253 for a particular angle or multiple different angles of a particular portion of the infant’s skin.
  • an infant skin profile may be created for when the plane of an infant’s forehead is approximately perpendicular to the line of sight of the streaming video camera (as exemplified by line-of-sight 381 in FIG. 3).
  • Another infant skin profile may be created for when the infant’s forehead is turned forty -five degrees to a particular side from the line-of-sight of the streaming video camera.
  • Infant skin profiles and related data may be stored to infant light absorption profile database 254, which may use a non-transitory processor-readable medium.
  • Infant light absorption profile database 254 may be incorporated as part of storage 215.
  • Each infant skin profile may indicate: a position and angle of a portion of an infant’s skin; a timestamp; one or more measured color values (e.g., an RGB value, hue, tint, saturation); and/or a corresponding background profile.
  • the corresponding background profile may have been captured as part of the same frame or frames as from which the infant skin profile was extracted.
  • Table 1 illustrates an example of several infant skin profiles:
  • each numerical indication of“infant skin position” corresponds to a different position and angular alignment of a particular portion of the infant’s skin (e.g., forehead) with the streaming video camera.
  • the timestamp indicates the date and time at which the one or more frames used to create the infant skin profile were collected.
  • “Color value” can represent an average color (e.g., in RGB) for the portion of the infant’s skin. Instead of an average value, multiple color values may be stored or a single value may be calculated using another form of calculation.
  • a corresponding background profile may indicate an identifier of the corresponding background profile, such as a file name or other unique identifier. It should be understood that Table 1 is merely an example; other embodiments of an infant skin profile may include additional and/or different data.
  • infant light absorption profile creator 253 may output the created infant skin profile to infant light absorption profile creator 253 and also to bilirubin analysis engine 256.
  • Infant light absorption profile creator 253 may store infant skin profiles for a variety of different dates, times, and lighting conditions.
  • each infant skin profile stored within infant light absorption profile creator 253 may be linked with a particular background model that was created based on the same frame or frames from which the infant skin profile was extracted. Therefore, the linked background model is representative of the same lighting conditions as the infant skin model.
  • Such an infant skin profile may be used for a future comparison having similar lighting conditions or for some other form of reference.
  • Background profile creator 252 may create a background model using one or more particular objects located within the background of the received portion of a frame. Infant skin detection engine 251 may transfer the portion of a frame that does not include infant skin to background profile creator 252. For one or more objects within a same frame (or group of frames), a background profile may be created. The background profile creator 252 may analyze objects having a known color profile and objects that are present in other background models for which a known color profile is not available but which can be compared between different background profiles.
  • a reference object having a color profile (a “known color”) that is accessible to background profile creator 252 may be identified within the streaming video camera’s field of view.
  • the video stream may be analyzed over an extended period of time, such as several hours or several days, to identify one or more objects that qualify to serve as a reference object.
  • Objects present in the video stream may be analyzed to determine whether: 1) the object in the video feed is underexposed or overexposed; 2) the object is glossy; 3) the object has enough detectable texture; 4) the object has not moved; and/or 5) the object is of a sufficient size.
  • the object may be required to satisfy at least these four conditions by not being under- or overexposed, by not being glossy, by having enough detectable texture, and/or by not moving within the field of view.
  • a reference object may serve as a reference to assess how the current lighting conditions in the video stream’s field-of-view is affecting the measured color of objects, including the infant’s skin.
  • the one or more reference objects may remain within the streaming video camera’s field of view for future reference (such as a crib, picture, particular toy, or reference color swatch).
  • an object that includes known colors may be present initially in a video feed, such as a color calibration pattern (e.g., a piece of paper or cardboard that has printed color swatches on it). Color values of the known colors may be stored or accessible by bilirubin analysis system 140. A lighting transform may be used to determine the difference between the known color value of one or more swatches on the color calibration card and the detected color in the video stream. Such an object may only be present during the initial setup process of the video stream for bilirubin detection and may be removed from the streaming video camera’s field-of- view following setup being completed.
  • bilirubin analysis engine 256 determine an absolute level of bilirubin in an infant’s blood. Otherwise, if a known reference color is not present within the environment, it may be possible to determine whether bilirubin levels are increasing, decreasing, or staying constant, but determining an absolute value may not be possible.
  • Background profile creator 252 may create a background profile for one or more objects that qualify as reference objects. Objects such as small toys, persons, pets, and clothing may be ignored.
  • a background profile may indicate: one or more objects, a color value corresponding to such objects (e.g., an RGB value, hue, tint, saturation); a timestamp, a background profile identifier, a corresponding infant skin profile identifier.
  • Table 2 illustrates an example of several background profiles:
  • identifier (such as a specific location within the frame) may be used.
  • a color value is measured. This color value may be an average of the measured color of the object or may correspond to a specific location (e.g., pixel) of the object. Due to different lighting conditions, the measured colors of different objects vary at different times. The timestamp indicates the date and time at which the one or more frames used to create the background profiles were collected.
  • Background profile identifier may indicate a unique identifier of the background profile, such as a file name or other unique identifier.
  • Corresponding infant skin profile identifier may indicate a unique identifier of the corresponding infant skin profile that was obtained from the same one or more frames. It should be understood that Table 2 is merely an example, other embodiments of an infant skin profile may include additional and/or different data.
  • background profile creator 252 may output the created background profile to background profile database 255 and, possibly, also to bilirubin analysis engine 256.
  • Background profile database 255 may store background profiles for a variety of different dates, times, and lighting conditions.
  • each background profile stored within background profile database 255 may be linked with a particular infant skin model that was created based on the same frame or frames from which the background profile was extracted. Such a background profile may be used for a future comparison having similar lighting conditions or for some other form of reference.
  • Bilirubin analysis engine 256 may perform processing on infant skin profiles and background profiles to determine a relative bilirubin level present in an infant’s blood (e.g., whether the infant’s bilirubin level is increasing, decreasing, or staying the same).
  • bilirubin analysis engine 256 may receive a recently created infant skin profile and background profile from infant light absorption profile creator 253 and background profile creator 252.
  • Bilirubin analysis engine 256 may retrieve one or more background profiles from background profile database 255 that matches or most-nearly matches the color values for the same one or more monitored background objects as present in the recently received background profile.
  • the infant skin profile corresponding to the retrieved background profile may be retrieved from infant light absorption profile creator 253.
  • the recently received infant skin profile may be compared directly with the infant-skin profile retrieved from infant light absorption profile creator 253 that was indicated as linked with the matching retrieved historic background profile from background profile database 255. Based on the comparison between the colors of the infant’s skin detected in the two infant skin profiles, it can be determined if the infant’s skin is absorbing more, less, or about the same amount of blue light, which is indicative of a bilirubin level in the infant’s blood. Further detail regarding how bilirubin analysis engine 256 may analyze the bilirubin level in the infant’s blood may be found detailed in relation to FIGS. 6-8.
  • a light having a known color profile (e.g., a light of the streaming video camera or another smart device, as previously detailed) is used to illuminate the field-of- view of the streaming video camera.
  • Bilirubin analysis system 140 may either trigger or track when the light is illuminated and unilluminated. Since the color profile of the light is known, the color delta between when background objects and the infant skin is illuminated by the light and not illuminated by the light can be measured.
  • Bilirubin analysis engine 256 may determine, using any or a multiple of the above methods, an absolute level of bilirubin within an infant’s blood or a trend in whether bilirubin is increasing, decreasing, or staying approximately consistent. The determination may be stored as part of bilirubin analysis storage 258. The determination may be stored in combination with a timestamp. [0049] Notification engine 257 may cause one or more notifications to be transmitted to an end use device linked with the same management account that was linked with the video feed received by stream processing engine 210.
  • a notification transmitted by notification engine 257 may indicate: an absolute amount of bilirubin; a trend in whether bilirubin levels are increasing, decreasing, or staying consistent; a comparison to a typical level or range of bilirubin for the infant’s age; and one or more recommendations or tips on how best to care for the infant due to bilirubin levels (e.g., treat infant normally, schedule a doctor’s visit, go to the emergency room).
  • Notifications may be sent periodically, such as once per day, and/or when a change in the infant’s bilirubin level is detected. For instance, if video frames of the infant are captured and it is determined that bilirubin levels have increased over the past 24 hours, a notification indicating as such may be sent to an end user linked with the management account that is linked with the video feed.
  • FIG. 3 illustrates an embodiment 300 of a room having a streaming video camera 304 capturing a field-of-view 380 including infant 301 and the background.
  • streaming video camera 340 has a field-of-view 380 illustrated by dotted lines.
  • the background can include all objects other than infant 301 or the skin of infant 301.
  • Particular background objects that are present in many frames may be tracked. For instance, picture 350, crib 370, and toys 360 may be present in all frames taken over a given time period.
  • Lamp 330 may be a movable desk lamp. Lamp 330 may produce artificial light. Lamp 330 may be manually turned on or off by a user. Alternately, lamp 330 may be a smart light that may be wirelessly controlled by another device.
  • streaming video camera 340 or a local or cloud-based server system in communication with streaming video camera 340 may be able to control whether lamp 330 is turned on (illuminated) or off (unilluminated). While lamp 330 may be a desk lamp as pictured, other forms of lights may have similar capabilities. For instance, a light may be fixed in location, such as a fixture installed in or on a ceiling or wall. A light may be a floor lamp that is moveable. While floor lamps and desk lamps may be moveable, once placed in a particular location, such lamps may tend to remain stationary for long periods of time. [0052] In some embodiments, one or more smart home devices, such as hazard detector 390, may be present within the room.
  • Hazard detector 390 may have an integrated light that can be turned on or off.
  • some hazard detectors have an integrated night light feature that activates a light when movement is detected in a darkened room. Such a light may be repurposed and used for illuminating the room for measuring bilirubin.
  • Streaming video camera 340 or a local or cloud-based server system in communication with streaming video camera 340 may be able to control whether lamp 330 is turned on (illuminated) or off (unilluminated), such as by transmitting a command to hazard detector 390 to turn on or off an on-board light.
  • Lights for which the color output is known may be particularly useful in illuminating content of the room.
  • a service provider providing bilirubin measurements may also manufacture or distribute hazard detector 390.
  • the color profile of the light of hazard detector 390 may be stored by the service provider. Therefore, when the light of hazard detector 390 is illuminated, the particular color output of the integrated light may be accessible and may be used in assessing the bilirubin level of infant 301.
  • the particular color output of lights not manufactured or distributed by the service provider may also be stored or otherwise accessible.
  • the make and model of the lighting element of lamp 330 may be provided by a user to the bilirubin analysis system, which may in turn perform a lookup of the color profile of the identified lighting element. If lamp 330 is a smart device, the bilirubin analysis system may be able to query the device to determine the type of lighting element and/or the color output profile of the lamp’s lighting element.
  • FIG. 4 illustrates an embodiment of a smart home environment in which a bilirubin level in an infant may be monitored.
  • a streaming video camera may be incorporated as part of a smart home environment. Additionally or alternatively, a streaming video camera may be incorporated as part of some other smart home device, such as those detailed in relation to smart home environment 400.
  • the smart home environment 400 includes a structure 450 (e.g., a house, office building, garage, or mobile home) with various integrated devices. It will be appreciated that devices may also be integrated into a smart home environment 400 that does not include an entire structure 450, such as an apartment, condominium, or office space. Further, the smart home environment 400 may control and/or be coupled to devices outside of the actual structure 450. Indeed, several devices in the smart home environment 400 need not be physically within the structure 450. For example, a device controlling a pool heater 414 or irrigation system 416 may be located outside of the structure 450.
  • a pool heater 414 or irrigation system 416 may be located outside of the structure 450.
  • “smart home environments” may refer to smart environments for homes such as a single-family house, but the scope of the present teachings is not so limited.
  • the present teachings are also applicable, without limitation, to duplexes, townhomes, multi-unit apartment buildings, hotels, retail stores, office buildings, industrial buildings, and more generally any living space or work space.
  • the customer may be the landlord with respect to purchasing the unit
  • the installer may be a local apartment supervisor
  • a first user may be the tenant
  • a second user may again be the landlord with respect to remote control functionality.
  • identity of the person performing the action may be germane to a particular advantage provided by one or more of the implementations, such identity should not be construed in the descriptions that follow as necessarily limiting the scope of the present teachings to those particular individuals having those particular identities.
  • the depicted structure 450 includes a plurality of rooms 452, separated at least partly from each other via walls 454.
  • the walls 454 may include interior walls or exterior walls.
  • Each room may further include a floor 456 and a ceiling 458.
  • Devices may be mounted on, integrated with and/or supported by a wall 454, floor 456 or ceiling 458.
  • the integrated devices of the smart home environment 400 include intelligent, multi-sensing, network-connected devices that integrate seamlessly with each other in a smart home network (e.g., 202 FIG. 2) and/or with a central server or a cloud-computing system to provide a variety of useful smart home functions.
  • a smart home network e.g., 202 FIG. 2
  • a central server or a cloud-computing system to provide a variety of useful smart home functions.
  • the smart home environment 400 may include one or more intelligent, multi-sensing, network-connected thermostats 402 (hereinafter referred to as“smart thermostats 402”), one or more intelligent, network-connected, multi-sensing hazard detection units 404 (hereinafter referred to as“smart hazard detectors 404”), one or more intelligent, multi-sensing, network-connected entryway interface devices 406 and 420 (hereinafter referred to as“smart doorbells 406” and“smart door locks 420”), and one or more intelligent, multi-sensing, network-connected alarm systems 422 (hereinafter referred to as“smart alarm systems 422”).
  • the one or more smart thermostats 402 detect ambient climate characteristics (e.g., temperature and/or humidity) and control an HVAC system 403 accordingly.
  • a respective smart thermostat 402 includes an ambient temperature sensor.
  • the one or more smart hazard detectors 404 may include thermal radiation sensors directed at respective heat sources (e.g., a stove, oven, other appliances, a fireplace, etc.).
  • a smart hazard detector 404 in a kitchen 453 includes a thermal radiation sensor directed at a stove/oven 412.
  • a thermal radiation sensor may determine the temperature of the respective heat source (or a portion thereof) at which it is directed and may provide corresponding black- body radiation data as output.
  • the smart doorbell 406 and/or the smart door lock 420 may detect a person’s approach to or departure from a location (e.g., an outer door), control doorbell/door locking functionality (e.g., receive user inputs from a portable electronic device 466-1 to actuate the bolt of the smart door lock 420), announce a person’s approach or departure via audio or visual means, and/or control settings on a security system (e.g., to activate or deactivate the security system when occupants go and come).
  • the smart doorbell 406 includes some or all of the components and features of the camera 418-1.
  • the smart doorbell 406 includes a camera 418-1, and therefore, is also called“doorbell camera 406” in this document.
  • Cameras 418-1 and/or 418-2 may function as a streaming video camera (similar to streaming video cameras 110 and 340 of FIGS. 1 and 3, respectively) and the streaming audio device detailed in relation to various embodiments herein.
  • Cameras 418 may be mounted in a location, such as indoors and to a wall or can be moveable and placed on a surface, such as illustrated with camera 418-2.
  • Various embodiments of cameras 418 may be installed indoors or outdoors. For performing bilirubin measurements, such cameras 418 may be placed near a location where the infant is occasionally located, such as a crib, playpen, bassinet, bed, etc.
  • the smart alarm system 422 may detect the presence of an individual within close proximity (e.g., using built-in IR sensors), sound an alarm (e.g., through a built-in speaker, or by sending commands to one or more external speakers), and send notifications to entities or users within/outside of the smart home network 400.
  • the smart alarm system 422 also includes one or more input devices or sensors (e.g., keypad, biometric scanner, NFC transceiver, microphone) for verifying the identity of a user, and one or more output devices (e.g., display, speaker).
  • the smart alarm system 422 may also be set to an armed mode, such that detection of a trigger condition or event causes the alarm to be sounded unless a disarming action is performed.
  • an alarm system may be linked with a service provider other than a provider of cameras 418. As such, remote services provided by the alarm system may be provided by an entity that does not provide the video and/or audio storage and analysis.
  • the smart home environment 400 includes one or more intelligent, multi-sensing, network-connected wall switches 408 (hereinafter referred to as“smart wall switches 408”), along with one or more intelligent, multi-sensing, network-connected wall plug interfaces 410 (hereinafter referred to as“smart wall plugs 410”).
  • the smart wall switches 408 may detect ambient lighting conditions, detect room-occupancy states, and control a power and/or dim state of one or more lights. In some instances, smart wall switches 408 may also control a power state or speed of a fan, such as a ceiling fan.
  • the smart wall plugs 410 may detect occupancy of a room or enclosure and control supply of power to one or more wall plugs (e.g., such that power is not supplied to the plug if nobody is at home).
  • the smart home environment 400 of FIG. 4 includes a plurality of intelligent, multi-sensing, network-connected appliances 412 (hereinafter referred to as“smart appliances 412”), such as refrigerators, stoves, ovens, televisions, washers, dryers, lights, stereos, intercom systems, garage-door openers, floor fans, ceiling fans, wall air conditioners, pool heaters, irrigation systems, security systems, space heaters, window AC units, motorized duct vents, and so forth.
  • an appliance may announce itself to the smart home network, such as by indicating what type of appliance it is, and it may automatically integrate with the controls of the smart home.
  • the smart home may also include a variety of non-communicating legacy appliances 440, such as old conventional washer/dryers, refrigerators, and the like, which may be controlled by smart wall plugs 410.
  • the smart home environment 400 may further include a variety of partially communicating legacy appliances 442, such as infrared (“IR”) controlled wall air conditioners or other IR-controlled devices, which may be controlled by IR signals provided by the smart hazard detectors 404 or the smart wall switches 408.
  • IR infrared
  • the smart home environment 400 includes one or more network-connected cameras 418 that are configured to provide video monitoring and security in the smart home environment 400.
  • the cameras 418 may be used to determine occupancy of the structure 450 and/or particular rooms 452 in the structure 450, and thus may act as occupancy sensors.
  • video captured by the cameras 418 may be processed to identify the presence of an occupant in the structure 450 (e.g., in a particular room 452).
  • Specific individuals may be identified based, for example, on their appearance (e.g., height, face) and/or movement (e.g., their walk/gait).
  • Cameras 418 may additionally include one or more sensors (e.g., IR sensors, motion detectors), input devices (e.g., a microphone for capturing audio), and output devices (e.g., a speaker for outputting audio).
  • the cameras 418 are each configured to operate in a day mode and in a low-light mode (e.g., a night mode).
  • the cameras 418 each include one or more IR illuminators for providing illumination while the camera is operating in the low-light mode.
  • the cameras 418 include one or more outdoor cameras.
  • the outdoor cameras include additional features and/or components such as weatherproofing and/or solar ray compensation.
  • the smart home environment 400 may additionally or alternatively include one or more other occupancy sensors (e.g., the smart doorbell 406, smart door locks 420, touch screens, IR sensors, microphones, ambient light sensors, motion detectors, smart nightlights 470, etc.).
  • the smart home environment 400 includes radio-frequency identification (RFID) readers (e.g., in each room 452 or a portion thereof) that determine occupancy based on RFID tags located on or embedded in occupants.
  • RFID readers may be integrated into the smart hazard detectors 404.
  • the smart home environment 400 may also include communication with devices outside of the physical home but within a proximate geographical range of the home.
  • the smart home environment 400 may include a pool heater 414 that communicates a current pool temperature to other devices within the smart home environment 400 and/or receives commands for controlling the pool temperature.
  • the smart home environment 400 may include an irrigation system 416 that communicates information regarding irrigation and/or receives control information for controlling such irrigation systems.
  • Smart home assistant 419 may have one or more microphones that continuously listen to an ambient environment. Smart home assistant 419 may be able to respond to verbal queries posed by a user, possibly preceded by a triggering phrase. Smart home assistant 419 may stream audio and, possibly, video if a camera is integrated, as part of the device, to a cloud-based host system 464 (which represents an embodiment of cloud-based host 200 of FIG. 2). In some embodiments, a user may pose a query to smart home assistant 419 about bilirubin levels (e.g., trends, absolute levels) measured in an infant monitored by at least one of cameras 418. [0070] By virtue of network connectivity, one or more of the smart home devices of FIG.
  • a user may communicate with a device using a computer (e.g., a desktop computer, laptop computer, or tablet) or other portable electronic device 466 (e.g., a mobile phone, such as a smart phone).
  • a webpage or application may be configured to receive communications from the user and control the device based on the communications and/or to present information about the device’s operation to the user.
  • the user may view a current set point temperature for a device (e.g., a stove) and adjust it using a computer.
  • the user may be in the structure during this remote communication or outside the structure.
  • users may control smart devices in the smart home environment 400 using a network-connected computer or portable electronic device 466.
  • some or all of the occupants e.g., individuals who live in the home
  • Such registration may be made at a central server to authenticate the occupant and/or the device as being associated with the home and to give permission to the occupant to use the device to control the smart devices in the home.
  • An occupant may use their registered device 466 to remotely control the smart devices of the home, such as when the occupant is at work or on vacation.
  • the occupant may also use their registered device to control the smart devices when the occupant is actually located inside the home, such as when the occupant is sitting on a couch inside the home. It should be appreciated that instead of or in addition to registering devices 466, the smart home environment 400 may make inferences about which individuals live in the home and are therefore occupants and which devices 466 are associated with those individuals. As such, the smart home environment may“learn” who is an occupant and permit the devices 466 associated with those individuals to control the smart devices of the home.
  • devices 402, 404, 406, 408, 410, 412, 414, 416, 418, 420, and/or 422 are capable of data communications and information sharing with other smart devices, a central server or cloud-computing system, and/or other devices that are network- connected.
  • Data communications may be carried out using any of a variety of custom or standard wireless protocols (e.g., IEEE 802.15.4, Wi-Fi, ZigBee, 6L0WPAN, Thread, Z-Wave, Bluetooth Smart, ISA100.5A, WirelessHART, MiWi, etc.) and/or any of a variety of custom or standard wired protocols (e.g., Ethernet, HomePlug, etc.), or any other suitable communication protocol, including communication protocols not yet developed as of the filing date of this document.
  • any of the smart devices that have on-board lights and are in the vicinity of one of cameras 418 may be activated and deactivated to adjust lighting conditions by either camera 418 or cloud-based host system 464.
  • a more accurate determination of the infant’s bilirubin level may be determined.
  • the color profile of the lights of the smart devices may be accessible to cloud-based host system 464. This color profile can be used to more accurately determine the bilirubin level of the infant.
  • the smart devices serve as wireless or wired repeaters.
  • a first one of the smart devices communicates with a second one of the smart devices via a wireless router.
  • the smart devices may further communicate with each other via a connection (e.g., network interface 460) to a network, such as the Internet.
  • a connection e.g., network interface 460
  • the smart devices may communicate with a cloud-based host system 464 (also called a cloud-based server system, central server system, and/or a cloud-computing system herein), which represents an embodiment of cloud-based host system 140 of FIGS. 1 and 2.
  • Cloud-based server system 464 may be associated with a manufacturer, support entity, or service provider associated with the smart device(s).
  • a user is able to contact customer support using a smart device itself rather than needing to use other communication means, such as a telephone or Internet-connected computer.
  • software updates are automatically sent from cloud-based server system 464 to smart devices (e.g., when available, when purchased, or at routine intervals).
  • the network interface 460 includes a conventional network device (e.g., a router), and the smart home environment 400 of FIG. 4 includes a hub device 480 that is communicatively coupled to the network(s) 462 directly or via the network interface 460.
  • the hub device 480 is further communicatively coupled to one or more of the above intelligent, multi-sensing, network-connected devices (e.g., smart devices of the smart home environment 400).
  • Each of these smart devices optionally communicates with the hub device 480 using one or more radio communication networks available at least in the smart home environment 400 (e.g., ZigBee, Z-Wave, Insteon, Bluetooth, Wi-Fi and other radio communication networks).
  • the hub device 480 and devices coupled with/to the hub device can be controlled and/or interacted with via an application running on a smart phone, household controller, laptop, tablet computer, game console or similar electronic device.
  • a user of such controller application can view the status of the hub device or coupled smart devices, configure the hub device to interoperate with smart devices newly introduced to the home network, commission new smart devices, and adjust or view settings of connected smart devices, etc.
  • the hub device extends capabilities of low capability smart devices to match capabilities of the highly capable smart devices of the same type, integrates functionality of multiple different device types - even across different communication protocols - and is configured to streamline adding of new devices and commissioning of the hub device.
  • hub device 480 further includes a local storage device for storing data related to, or output by, smart devices of smart home environment 400.
  • the data includes one or more of: video data output by a camera device, metadata output by a smart device, settings information for a smart device, usage logs for a smart device, and the like.
  • smart home environment 400 includes a local storage device 490 for storing data related to, or output by, smart devices of smart home environment 400.
  • the data includes one or more of: video data output by a camera device (e.g., cameras 418 or doorbell camera 406), metadata output by a smart device, settings information for a smart device, usage logs for a smart device, and the like.
  • a camera device e.g., cameras 418 or doorbell camera 406
  • metadata output by a smart device e.g., settings information for a smart device, usage logs for a smart device, and the like.
  • local storage device 490 is communicatively coupled to one or more smart devices via a smart home network (e.g., smart home network 202, FIG. 2).
  • a smart home network e.g., smart home network 202, FIG. 2.
  • local storage device 490 is selectively coupled to one or more smart devices via a wired and/or wireless communication network.
  • local storage device 490 is used to store video data when external network conditions are poor.
  • local storage device 490 is used when an encoding bitrate of cameras 418 exceeds the available bandwidth of the external network (e.g., network(s) 462).
  • local storage device 490 temporarily stores video data from one or more cameras (e.g., cameras 418) prior to transferring the video data to a server system (e.g., server system 464).
  • service robots 468 each configured to carry out, in an autonomous manner, any of a variety of household tasks.
  • the service robots 468 can be respectively configured to perform floor sweeping, floor washing, etc. in a manner similar to that of known commercially available devices such as the RoombaTM and ScoobaTM products sold by iRobot, Inc. of Bedford, Massachusetts.
  • Tasks such as floor sweeping and floor washing can be considered as“away” or “while-away” tasks for purposes of the instant description, as it is generally more desirable for these tasks to be performed when the occupants are not present.
  • one or more of the service robots 468 are configured to perform tasks such as playing music for an occupant, serving as a localized thermostat for an occupant, serving as a localized air
  • Such tasks can be considered as“human-facing” or“human-centric” tasks.
  • service robots may have one or more cameras and/or microphones that enable service robots 468 to stream video and/or audio to cloud-based host system 464 (and thus perform the functions of a streaming video camera similar to one of cameras 418).
  • a particular service robot 468 can be considered to be facilitating what can be called a“personal health-area network” for the occupant, with the objective being to keep the air quality in the occupant’s immediate space at healthy levels.
  • other health-related functions can be provided, such as monitoring the temperature or heart rate of the occupant (e.g., using finely remote sensors, near-field communication with on-person monitors, etc.).
  • a particular service robot 468 When serving as a localized hazard detector for an occupant, a particular service robot 468 can be considered to be facilitating what can be called a“personal safety-area network” for the occupant, with the objective being to ensure there is no excessive carbon monoxide, smoke, fire, etc., in the immediate space of the occupant.
  • Methods analogous to those described above for personal comfort-area networks in terms of occupant identifying and tracking are likewise applicable for personal health-area network and personal safety-area network embodiments.
  • the above-referenced facilitation of personal comfort- area networks, personal health-area networks, personal safety-area networks, and/or other such human-facing functionalities of the service robots 468 are further enhanced by logical integration with other smart sensors in the home according to rules-based inferencing techniques or artificial intelligence techniques for achieving better performance of those human-facing functionalities and/or for achieving those goals in energy-conserving or other resource-conserving ways.
  • the air monitor/purifier service robot 468 can be configured to detect whether a household pet is moving toward the currently settled location of the occupant (e.g., using on-board sensors and/or by data communications with other smart-home sensors along with rules-based inferencing/artificial intelligence techniques), and if so, the air purifying rate is immediately increased in preparation for the arrival of more airborne pet dander.
  • the hazard detector service robot 468 can be advised by other smart-home sensors that the temperature and humidity levels are rising in the kitchen, which is nearby the occupant’s current dining room location, and responsive to this advisory, the hazard detector service robot 468 will temporarily raise a hazard detection threshold, such as a smoke detection threshold, under an inference that any small increases in ambient smoke levels will most likely be due to cooking activity and not due to a genuinely hazardous condition.
  • a hazard detection threshold such as a smoke detection threshold
  • FIG. 5 illustrates an embodiment of a method 500 for non-invasively measuring bilirubin in blood, such as in the blood of an infant.
  • Method 500 may be performed using the systems and devices of FIGS. 1-4. More specifically, blocks of method 500 may be performed by bilirubin analysis system 140, which may reside at a cloud-based host system.
  • a video stream may be captured and transmitted to a bilirubin analysis system.
  • streaming video camera 110 may capture video and, possibly, audio, which is transmitted, via the Internet, to a cloud-based host system.
  • the cloud-based host system may perform multiple functions, including storing video for later retrieval.
  • One of the services provided by the cloud-based host system may include bilirubin analysis by a bilirubin analysis system.
  • the video stream may capture a field-of-view of a room in which a person, such as an infant, is occasionally located and for whom a bilirubin level is to be analyzed.
  • the streaming video camera may transmit some number of frames per second, such as 10 or 30 frames per second.
  • the video stream may be received by a bilirubin analysis system, such as bilirubin analysis system 140.
  • the bilirubin analysis system may be located locally to the streaming video camera or possibly integrated as part of the streaming video camera.
  • the video stream may not need to be transmitted to a cloud-based system (or any other form of remote system) for analysis. Rather, analysis may be performed locally.
  • a background model may be created using one or more portions of the frame.
  • the background model created at block 515 may be performed in accordance with the creation of a background profile detailed in relation to background profile creator 252.
  • the background profile may indicate information, such as color values, for one or more objects that are present in the background of the one or more frames and are being used as reference objects.
  • a background model may be retroactively created. That is, rather than waiting a period of time for the background of the video stream to be monitored, analyzed, and created prior to an infant being monitored for bilirubin, the infant may immediately start being monitored, but a determination may not be made on bilirubin levels until the background model has been created and used to analyze previously captured images of the infant. Image data related to the infant may be stored and not used for providing any output indicative of bilirubin levels until the background model has been successfully created. During the same time period that the infant is being monitored, objects in the background may be monitored and a background model may be created.
  • the captured images that represent the infant may be analyzed to determine if the captured images match the lighting conditions of the background model (by comparing the background model data to the same background objects in the images of the infant to be analyzed).
  • blocks 530-545 may be performed.
  • a portion of a frame or group of frames that contains infant skin may be detected. Portions of the frame or group of frames that does not include the infant may be considered background.
  • the infant skin may be identified using object recognition techniques, and/or facial recognition techniques, and/or detecting particular color ranges which are expected to correspond to infant skin.
  • a particular portion of an infant’s skin may be detected, such as chest or forehead.
  • only infant skin that is in a particular orientation in relation to the streaming video camera may be detected. For example, referring to FIG. 3, only forehead skin of infant 301 that is perpendicular to line of sight 381 with streaming video camera 340 may be detected at block 520. Having a consistent orientation of a portion of the infant skin with streaming video camera 340 can help ensure consistency of lighting across frames captured at different times.
  • an infant skin model may be created.
  • the infant skin model created at block 530 may be created as detailed in relation to infant light absorption profile creator 253.
  • the infant skin profile may indicate a particular color for the portion of the infant’s skin analyzed.
  • the background model created at block 525 may be mapped to the infant skin model created at block 530. It may be assumed that the ambient lighting that affects the colors of background objects in the background model may similarly affect the color of the infant’s skin in the infant skin model.
  • the approximate color temperature of 2700 K of the incandescent light may equally affect the detected coloring of the infant’s skin and the one or more background objects present in the background model.
  • an amount of blue light absorbed by the skin of the infant may be determined using the infant skin model created at block 530. For instance, the average blue light absorption over a patch of skin may be determined. For example, the blue value in a measured RGB value for each pixel corresponding to the patch of skin may be averaged together.
  • blocks 540 and 545 may be skipped. Since a goal of method 500 is to determine if bilirubin levels are relatively increasing, decreasing, or staying consistent over time, blocks 540 and 545 may not be executed a first time that method 500 is performed since a comparison needs to be made to determine if the amount of bilirubin in the infant’s blood is increasing, decreasing, or staying the same. [0088] At block 547, a determination may be made as to whether an additional measurement is to be performed. Measurements may be made over several days until a determination is made (either by the bilirubin analysis system, by a user, or by a doctor) that bilirubin monitoring is no longer needed.
  • method 500 may proceed to block 560. Otherwise, method 500 may proceed to block 550, at which an amount of time is waited before reanalyzing the amount of bilirubin in the infant’s skin. For example, the period of time to wait may be fifteen minutes, one hour, four hours, or one day, to name only a handful of examples. Longer or shorter durations of time for block 550 may be possible.
  • a determination may be made as to whether the lighting conditions match the lighting conditions used to create the background model at block 525.
  • the determination of block 550 may involve analyzing the color of the one or more reference objects in the video stream as being currently received. If the color of the reference objects is within a predefined threshold delta of the measured color indicated in the background model, method 500 may return to block 530. If not, method 500 may return to block 550 and continue to wait until the colors of the reference objects in the video feed match the colors of the reference objects indicated in the stored background model of block 525.
  • method 500 may proceed back to block 520.
  • the portion of a new frame present in the video feed that corresponds to infant skin may be determined.
  • a second (or subsequent) skin model for the infant may be created. This infant skin model may be directly comparable to each earlier determined infant skin model since the color of the background reference objects were determined to match, within a threshold range, the colors present in the background model of block 525.
  • a second amount of blue light absorbed by the skin of the infant may be determined using the infant skin model created at block 530. For instance, the average blue light absorption on the same patch of skin at which block 535 was first evaluated may be determined.
  • a determination may be made as to whether blue light absorption has increased or decreased by comparing the amounts of blue light absorption from each time that block 535 has been performed. If blue light absorption is increasing, this may be a sign that bilirubin levels are increasing. If blue light absorption is decreasing, this may be a sign that bilirubin levels are decreasing.
  • a notification may be output that indicates whether bilirubin levels appear to be increasing, decreasing, or staying constant. For example, twice per day, a message may be provided to a user linked to the management account with which the streaking video camera is linked that indicates the infant’s bilirubin state.
  • a chart, such as presented in FIGS. 9 and 10 may be provided, such as through an installed application on a mobile device, that indicates how the bilirubin level is changing relative to previous measurements.
  • FIG. 6 illustrates an embodiment of a method 600 for measuring bilirubin in blood using a lighting transform.
  • Method 600 may be performed using the systems and devices of FIGS. 1-4. More specifically, blocks of method 600 may be performed by bilirubin analysis system 140, which may reside at a cloud-based host system.
  • Blocks 605, 610, 615, 620, 630, 635, 640, 645, 647, and 650 may be performed as detailed in relation to the corresponding blocks of method 500 of FIG. 5.
  • a determination may be made as to whether the lighting conditions sufficiently match the lighting conditions used to create the background model at block 615.
  • the determination of block 650 may involve analyzing the color of the one or more reference objects in the video stream as being currently received in the video stream. If the color of the reference objects is within a predefined threshold delta of the measured color indicated in the background model, method 600 may proceed back to block 620, at which a portion of a recently- received frame that corresponds to infant skin is detected.
  • a lighting transform may be applied to the one or more video frames to be analyzed.
  • the lighting transform may be a form of color balancing by globally adjusting the intensities of colors (e.g., red, blue and green) within the image such that the reference objects in the received image match or match within a threshold the colors of the reference objects as indicated in the background model. That is, a color balancing adjustment may be made globally to the image in the red channel, green channel, and blue channel.
  • the infant’s skin has been color balanced to compensate for current lighting conditions, can then be used to accurately used to detect the portion of the frame corresponding to the infant’s skin at block 620, determine the amount of blue light absorption at block 635, and can be used for comparison to other blue light absorption measurements for use in determining if bilirubin levels are increasing or decreasing.
  • a lighting transform at block 650 may be performed to compensate for the current lighting conditions as compared to the lighting conditions used for the creation of the background model.
  • FIG. 7 illustrates an embodiment of a method 700 for measuring bilirubin in blood using interpolation between multiple background models.
  • Method 700 may be performed using the systems and devices of FIGS. 1-4. More specifically, blocks of method 700 may be performed by bilirubin analysis system 140, which may reside at a cloud-based host system. Blocks 705 and 710 may be performed as detailed in relation to the corresponding blocks of method 500 of FIG. 5.
  • multiple background models may be created. For instance, a first background model may be created when only artificial lighting lights objects within the room. A second background model may be created for when natural light is illuminating objects within the room. In order for the multiple background models to be created, the video stream may be analyzed for a significant period of time, such as several hours or days. Block 720 may be performed similarly to as detailed in relation to FIG. 5.
  • an infant skin model may be created.
  • the infant skin model created at block 730 may be created as detailed in relation to infant light absorption profile creator 253.
  • the infant skin profile may indicate a particular color for the portion of the infant’s skin analyzed.
  • the infant skin model may be linked to a particular background model of the background models created at block 715.
  • the linked background model may be indicative of the current lighting conditions when the infant skin model was created.
  • Blocks 730, 735, 740, 745, 747, and 750 may be performed as detailed in relation to blocks 530 through 550 of method 500.
  • a determination may be made if the lighting conditions determined based on the reference objects match, within a threshold, one of the created multiple background models. If yes, method 700 may proceed to block 730 at which an infant skin model is created that corresponds to the matching background model. If block 755 is determined in the negative, method 700 may proceed to block 760.
  • interpolation may be performed based on at least two of the background models and the lighting conditions currently observed based on the reference objects within the video feed to create an interpolated background model. Following interpolation being used to adjust the frame image to be analyzed, block 720 may be performed to locate infant skin within the frame.
  • the two or more background models may be used to create an interpolated background model.
  • the background model that was created based on natural lighting and the background model that was created based on artificial lighting may be used to create an interpolated background model.
  • the amount of blue light absorption determined at block 735 may be based on an interpolated background model determined using the multiple background models created at block 715.
  • the amount of blue light that is determined to be absorbed by the infant’s skin may be normalized based on either the particular background model that corresponds to the current lighting conditions or the interpolated background model that corresponds to the current lighting conditions.
  • FIG. 8 illustrates an embodiment of a method 800 for measuring bilirubin in blood using active scene illumination.
  • Method 800 may be performed using the systems and devices of FIGS. 1-4. More specifically, blocks of method 800 may be performed by bilirubin analysis system 140, which may reside at a cloud-based host system. Blocks 805 and 810 may be performed as detailed in relation to the corresponding blocks of method 500 of FIG. 5.
  • active scene illumination may be performed by causing a light in the environment of the streaming video camera to be activated. The light may be visible light, infrared, or some other form of
  • the light may be integrated as part of the streaming video camera or may be a separate smart light, such as smart home device 120.
  • smart illumination may be activated when there is no or little other lighting illuminating the field-of-view of the streaming video camera.
  • Blocks 820, 825, 830, 835, 840, 845, 847 and 850 may be performed as detailed in relation to blocks 520-550 of FIG. 5. Since the infant is only illuminated using the smart illumination, only a single background model may be created and used. Therefore, all blue light absorption measurements made of the infant’s skin may be compared with each other to determine if blue light absorption is increasing or decreasing. After block 820, and one or more frames have been captured that include the infant, the smart illumination may be turned off at block 823.
  • a determination may be made as to whether the lighting conditions are suitable for smart illumination to be illuminated. The determination may be based on whether there are no other sources of illumination illuminating the infant’s skin greater than a threshold brightness. Therefore, when the smart illumination is activated, the smart illumination source will be the lighting source that primarily illuminates the infant’s skin. If, at block 855, a determination is made that lighting conditions are acceptable, method 800 may proceed to block 815. Rather than an additional background model being created at block 825, the lighting of the reference objects may be confirmed to match the lighting of the reference objects indicated in the previously-created background model and method 800 may proceed to block 830. If, at block 855, a determination is made that the lighting conditions are not suitable for the smart illumination to be activated, such as if a nearby light is turned on, method 800 may return to block 850 to wait for a period of time.
  • FIG. 9 illustrates an embodiment 900 of an interface 910 on an end user device for reviewing an analyzing bilirubin levels determined using a bilirubin analysis system.
  • Interface 910 may be presented as part of a native application executed by an end user device.
  • relative changes in an infant’s bilirubin level are graphically illustrated on a day -by-day basis. This data may be obtained directly from bilirubin analysis engine 256 and/or retrieved from bilirubin analysis storage 258.
  • Interface 910 may also indicate a recommendation to an end user. As illustrated, since bilirubin levels are trending down, no action is needed. Referring to interface 1010 of embodiment 1000 of FIG. 10, bilirubin levels are trending up, thus a recommendation may be to contact the infant’s healthcare provider immediately. Interfaces 910 and 1010 may permit an end user to take various actions, such as sending the bilirubin measurements to the infant’s doctor. It should be understood that interfaces 910 and 1010 are examples of how bilirubin data created by bilirubin analysis system 140 can be presented. In other embodiments, such data may be presented via a webpage, numerically, or in some other form.
  • configurations may omit, substitute, or add various procedures or components as appropriate.
  • the methods may be performed in an order different from that described, and/or various stages may be added, omitted, and/or combined.
  • features described with respect to certain configurations may be combined in various other configurations. Different aspects and elements of the configurations may be combined in a similar manner. Also, technology evolves and, thus, many of the elements are examples and do not limit the scope of the disclosure or claims.
  • configurations may be described as a process which is depicted as a flow diagram or block diagram. Although each may describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. A process may have additional steps not included in the figure.
  • examples of the methods may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks may be stored in a non-transitory computer-readable medium such as a storage medium. Processors may perform the described tasks.

Landscapes

  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Physics & Mathematics (AREA)
  • Biomedical Technology (AREA)
  • Medical Informatics (AREA)
  • Veterinary Medicine (AREA)
  • Public Health (AREA)
  • Biophysics (AREA)
  • Pathology (AREA)
  • Engineering & Computer Science (AREA)
  • General Health & Medical Sciences (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Animal Behavior & Ethology (AREA)
  • Molecular Biology (AREA)
  • Surgery (AREA)
  • Optics & Photonics (AREA)
  • Chemical & Material Sciences (AREA)
  • Chemical Kinetics & Catalysis (AREA)
  • General Chemical & Material Sciences (AREA)
  • Dentistry (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Measuring And Recording Apparatus For Diagnosis (AREA)
  • Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)

Abstract

L'invention concerne diverses dispositions permettant de mesurer une quantité de bilirubine dans le sang. Une caméra vidéo à diffusion en continu permet de capturer un flux vidéo d'une partie d'une pièce pendant une certaine période. Un modèle d'arrière-plan peut être créé sur la base de conditions d'éclairage dans le flux vidéo. Une quantité d'absorption de lumière du sang présent dans la peau d'un nourrisson peut être déterminée à l'aide du modèle d'arrière-plan. L'état de la bilirubine du nourrisson peut ensuite être déterminé sur la base de la quantité d'absorption de lumière du sang dans la peau du nourrisson.
PCT/US2018/066753 2017-12-22 2018-12-20 Détection sans effraction de niveaux de bilirubine chez un nourrisson dans un environnement domestique intelligent WO2019126470A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201762609928P 2017-12-22 2017-12-22
US62/609,928 2017-12-22

Publications (1)

Publication Number Publication Date
WO2019126470A1 true WO2019126470A1 (fr) 2019-06-27

Family

ID=65244593

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2018/066753 WO2019126470A1 (fr) 2017-12-22 2018-12-20 Détection sans effraction de niveaux de bilirubine chez un nourrisson dans un environnement domestique intelligent

Country Status (1)

Country Link
WO (1) WO2019126470A1 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20210236844A1 (en) * 2018-04-27 2021-08-05 Koninklijke Philips N.V. A method and device for real time monitoring and prediction of bilirubin levels and associated notifications in neonates
EP4292515A1 (fr) * 2022-06-16 2023-12-20 Koninklijke Philips N.V. Surveillance sans contact d'un sujet

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140323832A1 (en) * 2013-04-24 2014-10-30 Selvaraj Thangaraj System and Method for non-invasive measurement of bilirubin in newborn and infants
US20150359459A1 (en) * 2013-03-12 2015-12-17 University Of Washington Through Its Center For Commercialization Systems, devices, and methods for estimating bilirubin levels
WO2017111606A1 (fr) * 2015-12-22 2017-06-29 Picterus As Détermination de bilirubine basée sur une image
US20170270365A1 (en) * 2014-07-07 2017-09-21 Google Inc. Systems and Methods for Categorizing Motion Events
US20170360340A1 (en) * 2013-07-08 2017-12-21 Koninklijke Philips N.V. System and method for extracting physiological information from remotely detected electromagnetic radiation

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150359459A1 (en) * 2013-03-12 2015-12-17 University Of Washington Through Its Center For Commercialization Systems, devices, and methods for estimating bilirubin levels
US20140323832A1 (en) * 2013-04-24 2014-10-30 Selvaraj Thangaraj System and Method for non-invasive measurement of bilirubin in newborn and infants
US20170360340A1 (en) * 2013-07-08 2017-12-21 Koninklijke Philips N.V. System and method for extracting physiological information from remotely detected electromagnetic radiation
US20170270365A1 (en) * 2014-07-07 2017-09-21 Google Inc. Systems and Methods for Categorizing Motion Events
WO2017111606A1 (fr) * 2015-12-22 2017-06-29 Picterus As Détermination de bilirubine basée sur une image

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20210236844A1 (en) * 2018-04-27 2021-08-05 Koninklijke Philips N.V. A method and device for real time monitoring and prediction of bilirubin levels and associated notifications in neonates
EP4292515A1 (fr) * 2022-06-16 2023-12-20 Koninklijke Philips N.V. Surveillance sans contact d'un sujet
WO2023241986A1 (fr) * 2022-06-16 2023-12-21 Koninklijke Philips N.V. Surveillance de sujet sans contact

Similar Documents

Publication Publication Date Title
US11967222B2 (en) Configuring a smart home controller
US10709335B2 (en) Infant monitoring system with observation-based system control and feedback loops
US10593184B2 (en) Baby monitoring with intelligent audio cueing based on an analyzed video stream
US20190205655A1 (en) Infant monitoring system with video-based temperature baselining and elevated temperature detection
US10621733B2 (en) Enhanced visualization of breathing or heartbeat of an infant or other monitored subject
US10262508B2 (en) Smart device with integrated conditional lighting
CN112567434B (zh) 智能门铃中的热管理
US10601604B2 (en) Data processing systems and methods for smart hub devices
US10649421B2 (en) Devices and methods for protecting unattended children in the home
US9685059B2 (en) Devices and methods for providing heat-source alerts
US11875659B2 (en) Privacy-preserving radar-based fall monitoring
US10610133B2 (en) Using active IR sensor to monitor sleep
US20170041524A1 (en) Devices and Methods for Repurposing IR Transmitters
EP3888344B1 (fr) Procédés et systèmes de coloration d'images infrarouges
WO2019126470A1 (fr) Détection sans effraction de niveaux de bilirubine chez un nourrisson dans un environnement domestique intelligent
US11259076B2 (en) Tactile launching of an asymmetric visual communication session

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18842607

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18842607

Country of ref document: EP

Kind code of ref document: A1