WO2013156908A1

WO2013156908A1 - Device and method for obtaining vital sign information of a living being

Info

Publication number: WO2013156908A1
Application number: PCT/IB2013/052880
Authority: WO
Inventors: Gerard De Haan; Vincent Jeanne; Ihor Olehovych Kirenko
Original assignee: Koninklijke Philips N.V.
Priority date: 2012-04-17
Filing date: 2013-04-11
Publication date: 2013-10-24

Abstract

The present invention relates to a device (1) and method for obtaining vital sign information of a living being (2). The proposed device (1) comprise a registration unit (3)for registering spatio-temporal variations of received light (4) in at least one wavelength interval and for generating an image signal (5) from said received light. The registration unit has an interface (6) for setting one or more parameters of said registration unit (3) related to the registration of spatio-temporal variations of received light. The device further comprises an image processing unit (7) for processing obtained image signals (5) and deriving vital sign information (8) of a living being represented in said image signals, and a control unit (9) analyzing derived vital sign information (8) and generating control information for controlling the setting of one or more parameters of said registration unit (3) to optimize the quality of the vital sign information derived from obtained image signals, said control unit (9) being coupled to said interface of said registration unit (3) to provide said control information to said interface for setting one or more parameters of said registration unit (3) according to said control information.

Description

Device and method for obtaining vital sign information of a living being

FIELD OF THE INVENTION

The present invention relates to a device and a corresponding method for obtaining vital sign information of a living being. Further, the present invention relates to a processor and a corresponding processing method for obtaining vital sign information of a living being from an image signal generated from spatio-temporal variations of received light registered in at least one wavelength interval. Still further, the present invention relates to a computer program.

BACKGROUND OF THE INVENTION

Recently, unobtrusive vital sign monitoring using a video camera, or remote PPG (photoplethysmography), has been demonstrated and found relevant for patient monitoring. Photo-Plethysmographic imaging is, for instance, described in Wim Verkruysse, Lars O. Svaasand, and J. Stuart Nelson, "Remote plethysmographic imaging using ambient light", Optics Express, Vol. 16, No. 26, December 2008. It is based on the principle that temporal variations in blood volume in the skin lead to variations in light absorptions by the skin. Such variations can be registered by a video camera that takes images of a skin area, e.g. the face, while processing calculates the pixel average over a manually selected region (typically part of the cheek in this system). By looking at periodic variations of this average signal, the heart beat rate and respiratory rate can be extracted.

Thus, the pulsation of arterial blood causes changes in light absorption. Those changes observed with a photodetector (or an array of photodetectors) form a PPG (photoplethysmography) signal (also called, among other, a pleth wave). Pulsation of the blood is caused by the beating heart, i.e. peaks in the PPG signal correspond to the individual beats of the heart. Therefore, a PPG signal is a heartbeat signal in itself. The normalized amplitude of this signal is different for different wavelengths, and for some wavelengths it is also a function of blood oxygenation.

Although regular video data have been shown to yield adequate vital signs (sometimes also called biometrical signals, such as heartbeat, respiration rate, Sp02 rate, etc.) in many cases, the image acquisition for challenging cases, like strong motion, low light levels, non-white illumination, needs further improvement. In cases with strong motion where it is desired to extract heartbeat from the skin of the face, for example the motion can cause the face to move in and out of the image if the camera is zoomed in too much. On the other hand if the camera is zoomed out too much there will not be much motion, but the number of pixels is limited so much that nonetheless a low signal to noise ratio results. In another example, the illumination of an environment is rather red (little green and blue) and consequently the signal to noise ratio in green and blue is poor.

WO 2011/055288 Al discloses a method of providing a combination of video data and metadata including obtaining a sequence of images captured by a video camera. At least one signal is extracted from the sequence of images, wherein each extracted signal characterizes local temporal variations in at least one of light intensity and color. At least one video compression technique is applied on image data of images from the sequence to obtain compressed video data. The extracted signals are extracted from images in a state prior to the application of the at least one compression technique to image data from those images. The compressed video data is provided with metadata for characterizing at least one process in a subject represented in at least part of the images, which process causes local temporal variations in at least one of color and intensity of light captured from the subject. The metadata is at least based on at least one of the extracted signals.

SUMMARY OF THE INVENTION

It is an object of the present invention to provide a device and a corresponding method as well as a processor and a corresponding processing method for obtaining vital sign information of a living being having a higher quality, in particular an increased signal-to- noise ratio (SNR) compared to known devices and methods.

In a first aspect of the present invention a device for obtaining vital sign information of a living being is presented comprising:

a registration unit for registering spatio-temporal variations of received light in at least one wavelength interval and for generating an image signal from said received light, said registration unit having an interface for setting one or more parameters of said registration unit related to the registration of spatio-temporal variations of received light, a image processing unit for processing obtained image signals and deriving vital sign information of a living being represented in said image signals,

a control unit for analyzing derived vital sign information and generating control information for controlling the setting of one or more parameters of said registration unit to optimize the quality of the vital sign information derived from obtained image signals, said control unit being coupled to said interface of said registration unit to provide said control information to said interface for setting one or more parameters of said registration unit according to said control information.

In a further aspect of the present invention a corresponding method for obtaining vital sign information of a living being is presented.

In a further aspect of the present invention a processing method for obtaining vital sign information of a living being from an image signal generated from spatio-temporal variations of received light registered in at least one wavelength interval is presented, comprising:

processing obtained image signals and deriving vital sign information of a living being represented in said image signals,

analyzing derived vital sign information and generating control information for controlling the setting of one or more parameters of said step of registering to optimize the quality of the vital sign information derived from obtained image signals, and

setting one or more parameters of said step of registering according to said control information, said one or more parameters of said step of registering being related to the registration of spatio-temporal variations of received light.

In yet another aspect of the present invention, there is provided a computer program which comprises program code means for causing a computer to perform the steps of the processing method when said computer program is carried out on a computer.

Preferred embodiments of the invention are defined in the dependent claims. It shall be understood that the claimed methods, processor, and computer program have similar and/or identical preferred embodiments as the claimed system and as defined in the dependent claims.

The present invention is based on the idea to control the setting of one or more parameters of said registration unit, e.g. video camera parameters in case of a video camera used as registration unit, to substantially optimize the quality of the extracted vital signs. To this end, instead of the regular settings of a registration unit that are aimed at realistic images, it is proposed according to the present invention to obtain and analyze vital sign information (which is generally derived from human skin in case of a person as living being). Based on the analysis of the vital sign information one or more settings of the registration unit are checked to confirm or modify them (or even originally set them as initial settings) so that the quality of the vital sign information becomes better, without taking into account to ensure that an image generated from the obtained image signals looks good for a viewer.

If it is, in parallel, also desired to construct an image that looks good, it is, of course, in parallel possible to obtain another set of image signals with other settings of the parameters of the registration unit that are optimized for construction such an image. The two sets of image signals may alternately be obtained if the settings are alternately changed in the same timing.

According to the above mentioned WO 2011/055288 Al the adjustment of camera parameters is working in parallel with the adjustment of compression parameters to achieve the best balance between the quality of the signal and compression ratio. In other words, both adjustments try to achieve the lowest bit-rate while preserving the reliability of the detected biometrical signals. In contrast, according to the present invention, the main aim for adjustment of the settings of parameters of the registration unit is the maximizing of the quality of the derived vital sign information.

In an embodiment of the present invention the vital sign extraction profits from an increase of the exposure and/or gain in green and blue, even if an image constructed from the obtained image signals looked quite unrealistic.

According to a preferred embodiment said registration unit comprises an imaging unit, in particular a camera, such as a video camera, RGB camera and/or infrared camera.

According to another preferred embodiment said registration unit is configured to generate an image signal for each of the at least one wavelength range. Thus, several image signals may be obtained for different wavelength ranges, and depending on the desired vital sign to be derived, the most appropriate one or more image signal may be used for deriving the vital sign information.

Preferably, said control unit is configured to generate control information for controlling the exposure time, gain and/or viewing angle of said registration unit. It has been shown that these parameters have a large influence on the quality of the vital sign

information derived from image signals. The adjustment of exposure time and gain is preferably done continuously to compensate for changes of ambient illumination or displacement of skin area. Those settings are generally not the best from e.g. picture quality point of view.

In an embodiment said control unit is configured to generate control information for increasing the exposure time and/or gain of said registration unit such that the image signal in the individual wavelength intervals, over a predetermined interval of time, comprises a minimum percentage of non-clipped pixels in a region of interest. By keeping a minimum (preferably constant) number of non-clipped pixels all non-clipped pixels create a sensing area for extraction of the vital sign information. Rapid or repetitive changes of the size of such sensing area might influence the consistency of the extracted vital signal information. Therefore, it is beneficial to preserve the size of the sensing area (number of non-clipped pixels) during any changes of lighting conditions or motion.

This embodiment aims at maximizing pixel value in sensing areas or regions of interest (e.g. part of a visible skin), even if all the remaining pixels (beyond of sensing area or region of interest) are clipped or too dark. In contrast, normally the settings of the parameters of a registration unit like a camera aim at optimizing all or the majority of pixels in the image, mostly from a visual quality point of view.

In an embodiment said control unit is configured to generate control information for increasing the exposure time and/or gain of said registration unit such that a combined pixel value, in particular an average of non-clipped pixels, in the individual wavelength intervals, over a predetermined interval of time (e.g. 1 second) is maximized. This further contributes to an optimization of the quality of the vital signs Clipped pixels generally introduce large fluctuations in the extracted signal during a motion. Therefore, it is advantageous to reduce the number of clipped pixels.

Said control unit is further configured in an embodiment to generate control information for increasing the viewing angle of the registration unit such that the number of skin pixels in the image signal, in particular averaged over a predetermined interval of time, is maximized. In addition or alternatively, other settings of one or more parameters of the registration unit may be adjusted such as to maximize the number of face pixels. Preferably, it is prevented that the face moves outside the image boundaries and the maximization of skin pixels should be achieved over at least 1 second, i.e. dynamically.

In an embodiment said control unit is configured to generate control information for keeping the average number of pixels in a predefined image area below a predetermined pixel number threshold. Sometimes, pixels from a background with a skin color, or pixels from skin areas of other subjects can be included by mistake in the predefined image area. Thus, it is preferred to remove pixels, which are outside the predefined area from an analysis.

Further, in an embodiment said control unit is configured to determine a quality metric from a derived vital sign information and to generate control information for controlling the setting of one or more parameters of said registration unit to optimize the quality metric. The goal of such a slow optimization is to avoid any additional high frequencies in the extracted vital sign information, which might be caused by rapid changes of settings of the registration unit.

Still further, in an embodiment said control unit is configured to generate control information for controlling the setting of one or more parameters of said registration unit only for a region of interest from which the vital sign information is derived. Thus, said parameters might be changed locally only for a certain spatial ROI (region of interest)or several ROIs, for instance around face or palm of a person).

Said ROI(s) may be selected either automatically, or manually, and not for the entire image. For this purpose the proposed device may further comprise a selection unit for automatically selecting said region of interest or allowing a manual selection of said region of interest.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other aspects of the invention will be apparent from and elucidated with reference to the embodiment(s) described hereinafter. In the following drawings

Fig. 1 shows an embodiment of a device for obtaining vital sign information of a living being according to the present invention.

DETAILED DESCRIPTION OF THE INVENTION

Fig. 1 shows a preferred embodiment of a device 1 for obtaining vital sign information of a living being 2, e.g. a patient in a hospital or an elderly person monitored in the bed at home, according to the present invention. The device 1 comprises a registration unit 3 for registering spatio-temporal variations of received light 4 in at least one wavelength interval and for generating an image signal 5 from said received light 4. The registration unit 3 is preferably an imaging unit for taking images, such as a video camera that substantially continuously or at regular intervals takes images of the living being 2.

The registration unit 3 has an interface 6 for setting one or more parameters of said registration unit 3 related to the registration of spatio-temporal variations of received light. While the registration unit 3 has a number of parameters that my by varied, the present invention is preferably directed to the setting of one or more of the exposure time, the gain and/or the viewing angle of the registration unit 3. The device 1 further comprises an image processing unit 7 that processes obtained image signals 4 and derives vital sign information 8 of a living being 2 represented in said image signals 5. The derivation of vital sign information, e.g. of the heartbeat, respiration signal, Sp02 value, etc., is generally known in the art, particularly in the field of remote photo-plethysmography, and shall not be explained here. The obtained vital sign information 8 is then output from the device 1, e.g. transmitted to a central monitoring station (e.g. a monitoring room of a nurse in a hospital) for display on a monitor, directly displayed next to the living being on a display, or transmitted to a remote control center for further processing and/or display.

Still further, the device 1 comprises a control unit 9 that analyzes derived vital sign information 8 and generates control information 10 for controlling the setting of one or more parameters of said registration unit 3 to optimize the quality of the vital sign

information 8 derived from obtained image signals 5. The control unit is thus coupled to said interface 6 of said registration unit 4 to provide said control information 10 to said interface 6 for setting one or more parameters of said registration unit 4 according to said control information 10.

Thus, the primary (and, in some applications, the only) interest of the device 1 according to the present invention is to obtain vital sign information having the best possible quality, meaning accuracy and reliability, without substantially increasing the hard- and software, processing time and other resources for implementing the device 1 compared to known devices for obtaining vital sign information of a living being.

Images obtained from the image signals 5 provided by the device 1 may not look visibly optimal, but may look distorted or unreal, but this is generally accepted in many applications. If, however, images having a visibly acceptable or optimal quality shall be obtained as well, a second set of image signals may be obtained in addition using other settings of the parameters of the registration unit 3, which may be optimized for deriving an image from the obtained image signals. This second set of image signals may be obtained in a separate time interval, or the settings of the parameters may be alternately switched between the settings optimal for deriving vital sign information and the settings optimal for image construction.

In a first practical implementation of the device 1 the control unit 10 is adapted to increase the exposure time and/or gain of the registration unit 3 such that the image signal in the individual wavelength intervals (e.g. in the visible wavelength intervals of red, green and blue visible light) over an interval of time (e.g. 1 second, but generally any lengths are usable, such as between 0.1 and 10 seconds) has a substantially fixed percentage of non-clipped pixels in a predetermined or selected region of interest (in particular a skin area) of the imaged area (field of view of the registration unit).

In a second practical implementation of the device 1 the control unit 10 is adapted to increase the exposure time and/or gain of the registration unit 3 such that a combined pixel value (e.g. average of non-clipping pixels) in the individual wavelength intervals, over an interval of time (e.g. 1 second), is maximized.

In a third practical implementation of the device 1 the control unit 10 is adapted to increase the viewing angle of the registration unit 3 such that the number of skin pixels in the imaged area, averaged over a period of time (e.g. 1 second), is maximized while keeping the average number of pixels in a predefined image boundary below a threshold value.

In a fourth practical implementation of the device 1 the control unit 10 is adapted calculate a quality metric from a derived vital sign and to slowly (time constant e.g. >1 second) adapt the setting of a parameter of the registration unit 3 to optimize the quality metric.

In a fifth practical implementation of the device 1 the control unit 10 is adapted to do the adaptation of parameters of the registration unit 3 for one from many possible regions of interest (ROI), particularly in case of monitoring of several subjects. For selecting the ROIs, either manually or automatically, a selection unit 11 (shown in broken line in Fig. 1) is provided. In this case an adjustment of camera parameters is implemented every time a new ROI is selected.

Finally, in an embodiment it is provided that for a number of the above adaptations, particularly the adaptation of the individual color gains and exposure time, it is kept track of the deviation from the 'standard' settings of these parameters for optimal image quality, so that the parameter (e.g. for the different colors) a correction is made in a later stage to have a good image quality again (for other purposes than vital sign extraction). This may lead to a compromise though, as the optimal setting for vital sign extraction may lead to many clipping pixels which cannot be corrected anymore.

The present invention may be applied in various applications. Heart rate, breathing rate, and Sp02 are very relevant factors in patient monitoring and home-healthcare where remote heart rate monitoring becomes more and more relevant. Further, the present invention may be applied to register heartbeat in fitness devices. It shall be noted that the image processing unit 7 and the control unit 9 are, in an embodiment, implemented on (the same or separate) processor(s) or computer(s), e.g. on a microprocessor, in the kind of a computer program which, when executed, carries out the steps of the proposed processing method.

While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive; the invention is not limited to the disclosed embodiments. Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims.

In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single element or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

A computer program may be stored/distributed on a suitable non-transitory medium, such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems.

Any reference signs in the claims should not be construed as limiting the scope.

Claims

CLAIMS:

1. Device for obtaining vital sign information of a living being (2), comprising:

a registration unit (3) for registering spatio-temporal variations of received light (4) in at least one wavelength interval and for generating an image signal (5) from said received light, said registration unit having an interface (6) for setting one or more parameters of said registration unit (3) related to the registration of spatio-temporal variations of received light,

an image processing unit (7) for processing obtained image signals (5) and deriving vital sign information (8) of a living being represented in said image signals,

a control unit (9) analyzing derived vital sign information (8) and generating control information for controlling the setting of one or more parameters of said registration unit (3) to optimize the quality of the vital sign information derived from obtained image signals, said control unit (9) being coupled to said interface of said registration unit (3) to provide said control information to said interface for setting one or more parameters of said registration unit (3) according to said control information.

2. Device as claimed in claim 1, wherein said registration unit (3) comprises an imaging unit, in particular a camera.

3. Device as claimed in claim 1, wherein said registration unit (3) is configured to generate an image signal for each of the at least one wavelength range.

4. Device as claimed in claim 1, wherein said control unit (9) is configured to generate control information for controlling the exposure time, gain and/or viewing angle of said registration unit (3).

5. Device as claimed in claim 1, wherein said control unit (9) is configured to generate control information for increasing the exposure time and/or gain of said registration unit (3) such that the image signal in the individual wavelength intervals, over a predetermined interval of time, comprises a minimum percentage of non-clipped pixels in a region of interest.

6. Device as claimed in claim 1, wherein said control unit (9) is configured to generate control information for increasing the exposure time and/or gain of said registration unit (3) such that a combined pixel value, in particular an average of non-clipped pixels, in the individual wavelength intervals, over a predetermined interval of time is maximized.

7. Device as claimed in claim 1, wherein said control unit (9) is configured to generate control information for increasing the viewing angle of the registration unit (3) such that the number of skin pixels in the image signal, in particular averaged over a

predetermined interval of time, is maximized.

8. Device as claimed in claim 7, wherein said control unit (9) is configured to generate control information for keeping the average number of pixels in a predefined image area below a predetermined pixel number threshold.

9. Device as claimed in claim 1, wherein said control unit (9) is configured to determine a quality metric from a derived vital sign information and to generate control information for controlling the setting of one or more parameters of said registration unit (3) to optimize the quality metric.

10. Device as claimed in claim 1, wherein said control unit (9) is configured to generate control information for controlling the setting of one or more parameters of said registration unit (3) only for a region of interest from which the vital sign information is derived.

11. Device as claimed in claim 10, further comprising a selection unit (11) for automatically selecting said region of interest or allowing a manual selection of said region of interest.

12. Method for obtaining vital sign information of a living being (2), comprising:

registering spatio-temporal variations of received light (4) in at least one wavelength interval, generating an image signal (5) from said received light,

processing obtained image signals (5) and deriving vital sign information (8) of a living being represented in said image signals,

analyzing derived vital sign information and generating control information (10) for controlling the setting of one or more parameters of said step of registering to optimize the quality of the vital sign information derived from obtained image signals, and setting one or more parameters of said step of registering according to said control information (10), said one or more parameters of said step of registering being related to the registration of spatio-temporal variations of received light.

13. Processor for obtaining vital sign information of a living being (2) from an image signal (5) generated from spatio-temporal variations of received light (4) registered in at least one wavelength interval, comprising:

a control unit (9) analyzing derived vital sign information (8) and generating control information (10) for controlling the setting of one or more parameters of said registration of received light to optimize the quality of the vital sign information derived from obtained image signals, said control unit (9) providing for setting one or more parameters of a registration of received light in said at least one wavelength interval according to said control information.

14. Processing method for obtaining vital sign information of a living being from an image signal generated from spatio-temporal variations of received light registered in at least one wavelength interval, comprising:

analyzing derived vital sign information (8) and generating control information (10) for controlling the setting of one or more parameters of said step of registering to optimize the quality of the vital sign information derived from obtained image signals, and

15. Computer program comprising program code means for causing a computer to carry out the steps of the method as claimed in claim 14 when said computer program is carried out on the computer.