Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
  • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B5/00—Measuring for diagnostic purposes; Identification of persons
  • A61B5/02—Detecting, measuring or recording for evaluating the cardiovascular system, e.g. pulse, heart rate, blood pressure or blood flow
  • A61B5/024—Measuring pulse rate or heart rate
  • A61B5/02416—Measuring pulse rate or heart rate using photoplethysmograph signals, e.g. generated by infrared radiation
• • G—PHYSICS
  • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
  • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
  • G16H30/00—ICT specially adapted for the handling or processing of medical images
  • G16H30/40—ICT specially adapted for the handling or processing of medical images for processing medical images, e.g. editing
• • G—PHYSICS
  • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
  • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
  • G16H40/00—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices
  • G16H40/60—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices
  • G16H40/67—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices for remote operation
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/10—Image acquisition modality
  • G06T2207/10016—Video; Image sequence
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/10—Image acquisition modality
  • G06T2207/10024—Color image
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/30—Subject of image; Context of image processing
  • G06T2207/30004—Biomedical image processing
  • G06T2207/30076—Plethysmography
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/30—Subject of image; Context of image processing
  • G06T2207/30196—Human being; Person
  • G06T2207/30201—Face

Definitions

• the present invention relates to a remote photoplethysmography device, a remote photoplethysmography method and a remote photoplethysmography system for registering a first image frame and a second image frame.
• Vital signs of a person for example the heart rate (HR), the respiration rate (RR) or the arterial blood oxygen saturation (SpO?), serve as indicators of the current state of a person and as powerful predictors of serious medical events. For this reason, vital signs are extensively monitored in inpatient and outpatient care settings, at home or in further health, leisure and fitness settings.
• HR heart rate
• RR respiration rate
• SpO? arterial blood oxygen saturation
• Plethysmography generally refers to the measurement of volume changes of an organ or a body part and in particular to the detection of volume changes due to a cardio-vascular pulse wave traveling through the body of a subject with every heartbeat.
• Photoplethysmography is an optical measurement technique that evaluates a time-variant change of light reflectance or transmission of an area or volume of interest.
• PPG is based on the principle that blood absorbs light more than surrounding tissue, so variations in blood volume with every heart beat affect transmission or reflectance correspondingly.
• a PPG waveform also called PPG signal
• PPG signal can comprise information attributable to further physiological phenomena such as the respiration.
• Conventional pulse oximeters for measuring the heart rate and the (arterial) blood oxygen saturation of a subject are attached to the skin of the subject, for instance to a fingertip, earlobe or forehead. Therefore, they are referred to as‘contact’ PPG devices.
• contact PPG is regarded as a basically non- invasive technique, contact PPG measurement is often experienced as being unpleasant and obtrusive, since the pulse oximeter is directly attached to the subject and any cables limit the freedom to move and might hinder a workflow.
• Non-contact, remote PPG (rPPG) devices also called camera-based device
• Remote PPG utilizes light sources or, in general radiation sources, disposed remotely from the subject of interest.
• a detector e.g., a camera or a photo detector
• Remote photoplethysmographic systems and devices are considered unobtrusive and well suited for medical as well as non-medical everyday applications.
• Remote PPG is e.g. disclosed in W. Wang, A. C. den Brinker, S. Stuijk, and G. de Haan,“Algorithmic principles of remote-PPG,” IEEE Transactions on Biomedical Engineering, vol.
• vital signs can be measured, which are revealed by minute light absorption changes in the skin caused by the pulsating blood volume, i.e. by periodic color changes of the human skin induced by the blood volume pulse.
• SNR signal-to-noise ratio
• Video Health Monitoring (Heart Rate, respiration rate, Sp02, actigraphy, delirium, etc.) is a promising emerging field. Its inherent unobtrusiveness has distinct advantages for patients with fragile skin, or in need of long-term vital signs monitoring, such as NICU patients, patients with extensive burns, or COPD patients who have to be monitored at home during sleep. In other settings such as in a general ward or emergency department, the comfort of contactless monitoring is still an attractive feature.
• a means for providing such a robust system for vital signs extraction is, for example, a combination of multi -wavelength channels to eliminate (motion-) distortions from a measured signal. Therefore, a system may have a multi -spectral camera equipped with Bayer filters to measure the skin at different wavelengths.
• Bayer filters are commonly available as RGB filters but less available as near infrared (NIR-) filters, which NIR wavelengths are particularly advantageous for pulse extraction.
• multiple monochrome cameras can be used having optical filters for predefined wavelengths or wavelength ranges.
• Such systems provide a cost effective alternative compared to multi-spectral camera systems and additionally allow a higher degree of freedom for wavelengths selection (e.g., 760 nm, 800 nm, 905 nm) compared to multi-spectral camera systems, since they are independent from the availability of, i.e., Bayer filters.
• multiple monochrome camera systems can be applied with narrow band filters which enable SpCh measurement.
• multi-spectral camera systems usually have cross-talk over bands that do not allow Sp0 2 measurement.
• Multiple monochrome camera systems commonly comprise two or more cameras spaced apart from one another viewing at a common region of interest. Since the two or more cameras are spaced apart, their optical paths are different when focusing the common region of interest. This leads to a displacement in position of the common region of interest when viewing at two image frames acquired from the two or more cameras, wherein the occurring phenomenon is commonly referred to as“parallax”. Since the significance of parallax on the measurement for vital signs extraction depends, among others, on focal length and a distance between the region of interest and the camera(s), it can still be considered as a significant challenge for the use of multiple monochrome cameras.
• image registration commonly comprises two stages, wherein in a first stage, a calibration is performed (e.g., for the first few frames of recording) in order to estimate a linear transformation model (such as translation, Euclidean, affine or homography) between image frames of different cameras. In a second stage, the estimated model may be applied to subsequent image frames to register said image frames to a reference image frame.
• a linear transformation model such as translation, Euclidean, affine or homography
• US 2018/0262685 A1 discloses a system that addresses the problem of parallax occurring when shooting, e.g., a 360° video using multiple lenses cameras. The system uses stitching of image frames from multiple cameras. A warp transformation is applied to determine a displacement of pixels of a border region of an acquired area.
• spatial and/or temporal smoothing is applied and the warp transformation is determined at multiple spatial scales.
• GAY-BELLILE Vincent, et al.“Image registration by combining thin-plate splines with a 3D morphable model”, in 2006 International Conference on Image Processing. IEEE, 2006.
• S. 1069-1072 discloses an image deformation model combining thin plate splines with 3D entities, a 3D control mesh and a camera.
• photoplethysmography device a remote photoplethysmography method and a remote photoplethysmography system that reduce the impact of parallax on the registration of image frames.
• a remote photoplethysmography device for registering a first image frame acquired by a first imaging unit and a second image frame acquired by a second imaging unit is presented, both the first and the second image frames depicting a common region of interest, the remote photoplethysmography device comprising a processing unit configured to measure a first pixel displacement between the first image frame and the second image frame, to correct the first pixel displacement according to spatial and/or temporal geometric constraints between the first imaging unit and the second imaging unit, and to register the first image frame with the second image frame based on the corrected first pixel displacement.
• a remote photoplethysmography system which comprises a first imaging unit configured to acquire a first image frame, a second imaging unit spaced apart from the first imaging unit and configured to acquire a second image frame, and a device for registering the first image frame and the second image frame according to the present invention.
• a corresponding remote photoplethysmography method a computer program which comprises program code means for causing a computer to perform the steps of the method disclosed herein when said computer program is carried out on a computer as well as a non-transitory computer-readable recording medium that stores therein a computer program product, which, when executed by a processor, causes the method disclosed herein to be performed.
• the present invention is based on the idea of using a non-linear adaptive image registration to solve the above-mentioned problems of conventional image registration.
• the processing unit involves measuring a displacement of pixels or a group of pixels between, for example, two different image frames preferably acquired by two different imaging units (i.e. cameras) at the same time, wherein the two image frames depict the same object or region of interest, respectively.
• the measured pixel displacement is preferably used to interpolate image frames in order to have the same alignment.
• spatial and/or temporal constraints of different image frames are used to restrict the measurement of pixel-to-pixel displacement or a displacement between a predefined group of pixels in order to smooth the interpolation.
• correction of the measured pixel displacement can either be based on solely spatial constraints, solely temporal constraints or both spatial and temporal (spatio-temporal) constraints jointly.
• a time window size may be defined.
• the term“time window size” may be understood as the time window length for a buffer of images.
• temporal image sequence may be set to at least two images in time (t t+1), but may be alternatively set to a couple of images in time (t t+N).
• the present invention has the advantage and strength that it allows non-linear registration between different imaging units.
• Such non-linear registration is adaptive to video content and can therefore be used to improve registration of image frames acquiring an object in motion.
• the image registration can be applied for scenes with depth changes or subject / object movements in an acquired scene (e.g., changes of a distance of the object to be measured to a respective imaging unit).
• the present invention shows a clear advantage in comparison to transformation based image registration (as e.g. disclosed in US 2018/0262685 Al), since transformation based image registration is linear and therefore has clear limitations when a scene with clear depth information and/or objects with 3D geometries is to be measured, as it is only effective for 2D planes.
• transformation based image registration uses estimation of a registration model which is not adaptive to video contents, since such a registration model is only valid for a subject (e.g., face) with the fixed distance-to-camera used for the model estimation. If the subject moves and changes its distance-to-camera during the measurement, the transformation based registration results are erroneous and may introduce, for example, color gradients and/or artifacts to vital signs extraction signal considered to be harmful for health signal extraction, especially for extraction methods such as blood volume pulse vector (PBV) method (as e.g. disclosed in G. de Haan, A. van Leest, "Improved motion robustness of remote-PPG by using the blood volume pulse signature", Physiol. Meas., vol. 35, no.
• PBV blood volume pulse vector
• the non-linear and adaptive image registration is considered to be advantageous.
• image registration according to the present invention can effectively reduce color artifacts normally caused by imperfect image registration that are well known for linear-based image registration.
• an integrated solution e.g., having a single optical path
• the present invention is considered to be beneficial, since a health monitoring system comprising the device according to the present invention and multiple imaging units, preferably multi-spectral NIR cameras, can be used as an effective alternative, since quality of image registration between the multiple imaging units can be significantly improved.
• wavelengths for calibration can be improved.
• the first image frame and the second image frame are acquired at a same point in time.
• the first and the second image frames used for registration are acquired by the two imaging units at the same time, but also that the two imaging units are synchronized to one another.
• the processing unit is configured to measure, as the first pixel displacement, a pixel -to-pixel displacement between pixels or a displacement between a group of pixels inside the region of interest.
• the measurement of the pixel displacement is preferably done by firstly selecting the first or the second image frame as a reference image frame. Secondly, a displacement of each pixel or a predefined group of pixels between the reference image frame and the non reference image frame is measured. Especially in cases where accuracy requirements are reduced, e.g. due to redundant information of measurement data, measurement of
• the processing unit is configured to measure the first pixel displacement based on a dense optical flow acquired for each individual pixel inside the region of interest or for a group of pixels inside the region of interest.
• dense optical flow refers to a pattern of an apparent motion of an object, a surfaces and/or edges between two image frames, either acquired at the same time by two different imaging units spaced apart from each other (parallax), or acquired at two different times by one imaging unit (i.e. when the object is in motion).
• dense optical flow can be defined as a distribution of apparent velocities of movement of pixels or a group of pixels between two different image frames.
• the dense optical flow is based on one of the Lukas Kanade flow, the Fameback flow, the Horn-Schunck flow, the block-matching flow, the deep-nets flow and/or the 3DRS flow.
• optical flow measurement methods differ in terms of efficiency (e.g. 3DRS is fast in computation).
• 3DRS is fast in computation.
• some of the above-mentioned optical flow measurement methods are more applicable to plain regions in image frames having fewer textures; others (namely more advanced/recent dense optical flow measurement method) are more applicable to featureless image regions.
• the processing unit is further configured to analytically calculate a second pixel displacement based on the spatial geometric constraints and/or the temporal geometric constraints and to smooth the first pixel displacement by calculating a mean value of the first pixel
• the value of the first pixel displacement can be refined based on predetermined spatial and/or temporal constraints, resulting in improved image registration.
• imprecision of registered pixels or a group of pixels in the region of interest can be minimized which improves evaluation accuracy of health related parameters.
• the processing unit is further configured to analytically calculate a second pixel displacement based on the spatial geometric constraints and/or the temporal geometric constraints, to detect outliners in the measured first pixel displacement by comparing said first pixel displacement with the second pixel displacement and to correct the first pixel displacement by rejecting the detected outliners.
• outliners resulting from imprecise registration i.e. caused by parallax of subject motion, can be rejected and therefore do not cause measurement inaccuracy in the measurement of health parameters.
• the outliners that have been removed are not used in the analysis of the health parameters.
• an average value i.e.
• the processing unit is configured to downscale the first image frame and the second image frame.
• the term“downscaling” refers to downsize the resolution of an image frame, for example resizing the image frame from 1240x720 pixels to 640x480 pixels.
• downscaling is done by spatial averaging of pixels.
• Advantages of downscaling are noise reduction (i.e. reduction of a camera sensor noise) and a generation of a more stable pixel representation (e.g. super pixel). Downsizing can be considered as similar to a block-to-block (group of pixels) local registration, rather than to a pixel-to-pixel local registration.
• the processing unit is configured to upscale the first pixel displacement.
• the considered image frame is firstly downscaled (e.g. from 1240x720 pixels to 640x480 pixels). Afterwards displacement vectors of downsized pixels or a group of downsized pixels are estimated for the downsize image frame (640x480 pixels) and the estimated displacement vectors are upscaled by multiplying them with a ratio (from 640x480 pixels to 1240x720 pixels), respectively. Then, the image frame using upscaled vectors in the original resolution is registered. Especially, the step of downscaling improves the speed of dense optical flow measurement and thus enhances overall registration efficiency.
• the spatial geometric constraints are based on predetermined geometric constraints between the first imaging unit and the second imaging unit.
• the spatial constraints may be measured using intrinsic parameters of the imaging units (i.e. imaging unit position, view angle, distance, etc.) and/or based on a measurement of image content.
• comer points or features represented in the image content may be used for referencing in order to derive spatial constraints.
• a view relation of a 3D object acquired by a multi-camera system having at least two cameras which are spaced apart from one another (and thus having a stereo vision on the 3D object) may be used as a spatial constraint.
• a view relation of 3D motion in such a multi-camera system may be used as a temporal constraint.
• the first imaging unit included in the system is a monochrome camera and/or a multi -spectrum camera and the second imaging unit included in the system is a monochrome camera and/or a multi-spectrum camera.
• a system may comprise one monochrome camera and one multi spectrum camera, wherein said cameras are spaced apart from one another.
• the monochrome camera may be configured to acquire a first wavelength or wavelength range
• the multi-spectrum camera may be configured to acquire both a second wavelength or wavelength range and a third wavelength or wavelength range.
• three different wavelengths can be measured by two different cameras which lead to a reduction of cost and overall system’s complexity.
• the first imaging unit included in the system is configured to acquire a first wavelength or wavelength range in the visible or infrared wavelength range and the second imaging unit included in the system is configured to acquire a second wavelength or wavelength range, different from the first wavelength or wavelength range, in the visible or infrared wavelength range.
• both the first and the second wavelength or wavelength range may be a NIR wavelength or wavelength range.
• the system further comprising a health parameter extraction unit configured to extract vital signs of a subject based on the registered image frame.
• the registered image frame obtained by the health parameter extraction unit results from the registration method according to the present invention which is performed by the device.
• the impact of parallax on vital signs extraction can be reduced.
• especially vital signs extraction from image frames of 3D subjects having depth information shows higher accuracy compared to state of the art vital signs extraction, i.e. based on
• Fig. 1 shows a schematic diagram of a system according to the present invention
• Fig. 2 shows a flowchart of a method according to the present invention
• Figs. 3A, 3B, 3C, 3D show a system setup (Fig. 3A) and resulting image frames (Figs. 3B, 3C, 3D) acquired by a system according to the present invention
• Fig. 4 shows a schematic diagram of a first group of image frames acquired by a system according to the present invention
• Figs. 5 A, 5B show a schematic diagram of a second group of image frames acquired by a system according to the present invention
• Fig. 6 shows a schematic diagram of a third group of image frames acquired by a system according to the present invention.
• Figs. 7A, 7B, 7C show image frames with no image registration (Fig. 7A), transformation based image registration (Fig. 7B) and image registration according to the present invention (Fig. 7C);
• Figs. 8A, 8B, 8C show image frames with no image registration (Fig. 7A), transformation based image registration (Fig. 7B) and image registration according to the present invention (Fig. 7C); and
• Figs. 9A, 9B show a comparison of a state of the art registration (Fig. 9A) and image registration according to the present invention (Fig. 9B).
• Fig. 1 shows a schematic diagram of an embodiment of a system 100 according to the present invention.
• the system 100 comprises a first imaging unit 110 configured to acquire a first frame 120 and a second imaging unit 130 configured to acquire a second image frame 140.
• the system 100 further comprises a device 150 for registering the first image frame 120 acquired by the first imaging unit 110 and the second image frame 140 acquired by the second imaging unit 130.
• the imaging units 110, 130 may also be referred to as camera-based or remote PPG sensors.
• Both image frames 120, 140 include information used to determine physiological information indicative of at least one vital sign of the subject 170.
• the subject 170 may be a patient, in this example a patient lying in a bed 180, e.g. in a hospital or other healthcare facility, but may also be a neonate or premature infant with very sensitive skin in NICU’s, e.g. lying in an incubator, a patient with damaged (e.g. burned) skin or a person at home or in a different environment.
• NICU very sensitive skin
• the first imaging unit 110 is a first camera and the second imaging unit is a second camera.
• the first camera 110 is a monochrome camera and the second camera 130 is a multi -spectrum camera.
• both the first and the second imaging unit 110, 130 may be a monochrome camera and/or a multi -spectrum camera.
• the first imaging unit 110 (the first camera 112) is configured to acquire a first wavelength (such as red light at 700 nm) or wavelength range (such as red light from 680 nm to 720 nm) or infrared wavelength range (above 790 nm), whereas the second imaging unit 130 (the second camera 130) is configured to acquire a second wavelength (such as green light at 550 nm) or wavelength range (such as green light from 530 nm to 570 nm).
• the second wavelength or wavelength range is preferably different from the first wavelength or wavelength range.
• the system may comprise more than two imaging units spaced apart from one another.
• the system may further comprise a third imaging unit depicting the common region of interest 160 of a subject 170.
• the third imaging unit is preferably configured to acquire a third image frame.
• the third imaging unit is configured to acquire a third wavelength or wavelength range which is preferably different from the first and the second wavelength or wavelength range.
• Both the first camera 110 and the second camera 130 preferably include a suitable photosensor for (remotely and unobtrusively) capturing image frames (such as the first image frame 120 and the second image frame 140) of the region of interest 160 of the subject 170, in particular for acquiring a sequence of image frames of the subject 170 over time, from which photoplethysmography signals can be derived.
• the image frames captured by the cameras 110, 130 may particularly correspond to a video sequence captured by means of an analog or digital photosensor, e.g. in a (digital) camera.
• Such cameras 110, 130 usually includes a photosensor, such as a CMOS or CCD sensor, which may also operate in a specific spectral range (visible, IR) or provide information for different spectral ranges.
• the cameras 110, 130 may provide an analog or digital signal.
• the image frames 120, 140 include a plurality of image pixels having associated pixel values. Particularly, the image frames 120, 140 include pixels representing light intensity values captured with different photosensitive elements of a photosensor. These photosensitive elements may be sensitive in a specific spectral range (i.e. representing a specific color or pseudo-color (in NIR)).
• the image frames 120, 140 include at least some image pixels being representative of a skin portion of the subject 170. Thereby, an image pixel may correspond to one photosensitive element of a photo-detector and its (analog or digital) output or may be determined based on a combination (e.g. through binning) of a plurality of the photosensitive elements.
• the system 100 may further comprise a light source (also called illumination source), such as a lamp, for illuminating the region of interest 160, such as the skin of the subject’s 170 face (e.g. part of the cheek or forehead), with light, for instance in predetermined wavelengths or wavelength ranges (e.g. in the red, green and/or infrared wavelength range(s)).
• a light source also called illumination source
• the region of interest 160 such as the skin of the subject’s 170 face (e.g. part of the cheek or forehead)
• predetermined wavelengths or wavelength ranges e.g. in the red, green and/or infrared wavelength range(s)
• the light reflected from said region of interest 160 in response to said illumination may be detected by the cameras 110, 130.
• no dedicated light source is provided, but ambient light is used for illumination of the subject 170. From the reflected light, only light in a number of desired wavelength ranges (e.g. green and red or infrared light, or light in a sufficiently
• the cameras 110, 130 may be applied with optical filters which are preferably different, though their filter bandwidth can be overlapping. It is sufficient if their wavelength-dependent transmission is different.
• the device 150 comprises a processing unit 190 which may be a processor of a computational device, system-on-a-chip or any other suitable unit for data processing.
• the processing unit 190 according to the embodiment shown in Fig. 1 is located inside the device 150, but may also be located outside, i.e., spaced apart from the device 150 and connected though one or more cables or wirelessly to device 150.
• the processing unit 190 is configured to measure a first pixel displacement 200 between the first image frame 120 and the second image frame 140 and thereby executes step SI 00 of the method according to another aspect of the present invention (see Fig. 2). Furthermore, the processing unit 190 is configured to correct the first pixel displacement 200 according to spatial and/or temporal geometric constraints between the first imaging unit 110 and the second imaging unit 130, thereby executing step S200 of the method according to the present invention. Then, the processing unit 190 is configured to register the first image frame 120 with the second image frame 140 based on the corrected first pixel displacement 200 by executing step S300 of the invention method (see Fig. 2).
• the method may comprise one or more intermediate steps which may precede or follow the above steps SI 00, S200, S300, respectively.
• information included in the image frames namely pixel based or group of pixels based color or grayscale information, may be extracted from image frames in time sequence. Therefore, a health parameter extraction unit 210 may be connected to the device 150 via one or more cables or wirelessly or may be integrated in the device 150.
• the health parameter extraction unit 210 may be preferably configured to extract one or more health related parameters from registered successive images frames (such as the image frames 120, 140).
• a system 100 as illustrated in Fig. 1 may, e.g., be located in a hospital, healthcare facility, elderly care facility or the like. Apart from the monitoring of patients, the present invention may also be applied in other fields such as neonate monitoring, general surveillance applications, security monitoring or so-called lifestyle environments, such as fitness equipment, a wearable, a handheld device like a smartphone, or the like.
• the uni- or bidirectional communication between the device 150 and the imaging units 110, 130 may work via a wireless or wired communication interface.
• Other embodiments of the present invention may include a device 150, which is not provided stand-alone, but integrated into at least one of the imaging units 110, 130.
• contactless monitoring may be more convenient than monitoring with contact sensors which is still used in a general ward or a triage in an emergency department.
• contactless monitoring may be applicable for monitoring of automotive drivers as well as for sleep monitoring, wherein in the latter, especially NIR- based monitoring, preferably multi-spectral NIR-based monitoring, may be applied to improve robustness of vital signs extraction.
• Fig. 3 A shows a schematic diagram of a second embodiment the system 100 further comprising a third imaging unit 220.
• the device 150 is integrated in the first imaging unit 110.
• the third imaging unit is configured to acquire a third image frame 230 depicting the common region of interest 160.
• the second and the third imaging unit 130, 220 are communicatively connected to the first imaging unit 110.
• the third imaging unit 220 is a camera 220.
• the system 100 setup represents a multi spectrum camera setup consisting of three cameras 110, 130, 220 that are spaced apart from one another depicting the same region of interest 160 viewing at the same direction (see in Figs. 3B, 3C, 3D).
• the cameras 110, 130, 220 are mono-chrome cameras with optical filters to sample desired wavelengths (which may be three different NIR
• the region of interest of the subject / object appearing in the image frames 120, 140, 230 acquired by the cameras 110, 130, 220 have displacement in their position with respect to one another. This displacement is commonly referred to as“parallax”.
• the significance of the parallax depends on the used focal length and the distance between the subject 170 and camera 110, 130, 220, respectively.
• the proposed nonlinear adaptive camera registration with spatio-temporal geometric constraints which is performed by the device 150 aims to improve the registration of the image frames 120, 140, 230 to eliminate said parallax problem.
• the nonlinear adaptive image registration is explained in detail referring to Figs. 4, 5A, 5B and 6.
• the image frames 120, 140, 230 have been acquired, for example, by a system according to Fig. 3 A.
• To enable the nonlinear registration preferably being a pixelwise registration or a registration of a predefined group of pixels across multiple cameras (here three cameras 110, 130, 220), is performed, wherein each pixel or group of pixels has its own registration in a new image frame.
• the image frames acquired by the central camera is taken as reference image frames.
• the image frame 120 acquired by the first imaging unit 110 is the reference image frame.
• new image frames are created.
• the camera 110 placed in the central position is selected as a reference camera, since it has the shortest distance to the subject 170 compared to the cameras 130, 220.
• the first pixel displacement 200 is measured which may be a pixel-to-pixel displacement between pixels or a displacement between a group of pixels.
• the first pixel displacement is measured, for example, between the first image frame 120 and the second image frame 140.
• a pixel displacement may be measured between the first image frame 120 and the third image frame 230.
• the first pixel displacement is measured between two pixels of the first image frame 120 relative to the second and third image frame 140, 230, respectively.
• the first pixel displacement may be measured by dense optical flow which results in motion vectors for pixels or group of pixels between the spatial and/or temporal image frames:
• D DOF(Iref, Inonref), where DOF(.) denotes the dense optical flow, I ref the reference image frame (i.e. image frame 120), I nonref the non-reference image frames (i.e. image frames 140, 230) and D the first pixel displacement 200, wherein D is used to correlate/interpolate the non-reference image frames 140, 230 in order to register them with the reference image frame 120: Ireg IntCrp(Inonicf, D), where Interp(.) denotes the interpolation/correlation and I reg the registered image.
• the pixel- based interpolation is highly nonlinear for image transformation and dense optical flow measurement and interpolation are performed for each individual image frame.
• the registration is adaptive to video contents and robust to scenes having depth changes or object position changes (e.g., distance-to-camera changes) during monitoring.
• the first pixel displacement is corrected according to spatial and/or temporal geometric constraints. These constraints are applied as a post processing step of the“raw” dense optical flow results described above. Since the setup (e.g., camera position) is preferably fixed during the measurement, parallax- induced pixel displacement across the cameras 110, 130, 220 depends on predefined geometric relationships (e.g., epipolar geometry). Thereby, epipolar geometry considers, for example, the geometry of stereo vision, wherein two cameras view a 3D scene from two distinct positions. In such a setup, there are a number of geometric relations between 3D points and their projections onto the 2D image frame leading to constraints between these image points.
• epipolar geometry considers, for example, the geometry of stereo vision, wherein two cameras view a 3D scene from two distinct positions.
• a second pixel displacement can be analytically calculated. Based on the second pixel displacement, the first pixel
• the displacement 200 may preferably be smoothed by calculating a mean value of the first pixel displacement 200 and the second pixel displacement.
• the second pixel displacement may be used to detect outliners in the measured first pixel displacement 200 by comparing said first pixel displacement 200 with the second pixel displacement and to correct the first pixel displacement 200 by rejecting the detected outliners.
• D’I 2 DI 3 - D2 3 (analytic solution)
• D”I 2 (DI 2 + D’I 2)/2 (smoothed solution)
• D I 2, D2 3, D I 3 denote the (group of) pixel displacement from camera 110 to 130, camera 130 to 220, and camera 110 to 220, respectively.
• D I 2 is the solution measured by dense optical flow
• D’ I 2 is the analytic solution deducted from D I 3 and D2 3
• D” I 2 is the smoothed solution resulting from the mean value between D I 2 and D’ I 2.
• it may be derived, i.e. from further metrics (not described here), that instead of the“smoothed solution” the“measured” or“analytic” solution may be more appropriate.
• D’ I 2 it is possible to use D’ I 2 to restrict measurement outliers in D I 2.
• temporal geometric constraints may be used to smooth the measurement of the first pixel displacement 200 (see Fig. 5B) which may be expressed by:
• T2 i denotes the pixel displacement from the second camera 130 at time t to the first camera 110 at time t+1 and T2 2 the pixel displacement from the first camera 110 at time t to the first camera 110 at time t+1.
• spatial and temporal geometric constraints are applied simultaneously (see Fig. 6).
• all pixels of a common region of interest of an object/subject are connected in space and time for globally optimizing / correcting the pixel displacement leading to a highly robust and smooth image registration.
• spatio-temporal geometric constraints Another benefit of using spatio-temporal geometric constraints is that the pixel motions (due to subject body motion or camera movement) between adjacent frames can be reduced.
• Figs. 7A, 7B, 7C show three image frames depicting a common region of interest 160 which is in this case a part of a face of a patient how’s vital signs are to be measured, wherein a distance between the at least two imaging units 110, 130 and the region of interest 160 of the subject 170 is short ( ⁇ 1 m) resulting in a large parallax.
• Figs. 7A, 7B, 7C show three image frames depicting a common region of interest 160 which is in this case a part of a face of a patient how’s vital signs are to be measured, wherein a distance between the at least two imaging units 110, 130 and the region of interest 160 of the subject 170 is short ( ⁇ 1 m) resulting in a large parallax.
• Figs. 7A, 7B, 7C show three image frames depicting a common region of interest 160 which is in this case a part of a face of a patient how’s vital signs are to be measured, wherein a distance between the
• FIG. 8 A, 8B, 8C show three image frames depicting a common region of interest 160 which is in this case a part of a face of another patient, wherein a distance between the at least two imaging units 110, 130 and the region of interest 160 of the subject 170 is large (5-6 m) resulting in a small parallax (i.e., where parallax is reduced manually).
• Fig. 7A and Fig. 8A both show an unregistered image frame in which strong blurriness is recognizable due to parallax. This leads, in particular when large parallax is apparent (see Fig. 7 A), to the fact that, for example, facial contours of the subject 170 are not delimitable as sharp edges / contours. Thus, the region of interest 160 may not be clearly delimited which may lead to an erroneous evaluation of health related parameters.
• Fig. 7B and Fig. 8B show a case where state of the art transformation based image registration has been performed resulting in a reduction of parallax based blurriness. However, especially when large parallax is apparent (see Fig. 7B), the state of the art registration still shows a remaining blurriness at an edge area of the region of interest 160 which still results in an incorrect evaluation of health related parameters, especially when the these areas include depth information.
• image registration has been performed according to an embodiment of the registration method according to the present invention resulting in an elimination of parallax based blurriness.
• image registration is performed resulting in sharp edges of a depicted region of interest.
• the improvement is particularly visible when the subject 170 changes the position or rotates the head during the monitoring, as the distance-to- camera or the face 3D geometry may change which renders the linear model used by the existing method invalid (i.e., transformation model is usually estimated upfront the measurement).
• Fig. 9A shows vital signs extraction based on a state of the art registered image frames.
• Fig. 9B shows vital signs extraction based on inventively registered image frames.
• pulse-rate and Sp02 have been extracted as exemplary vital signs and the vital signs extraction method is the same for both cases.
• curve progression is more stable when registration has been performed according to the present invention. The higher stability results from the fact that Sp02 measurement is based on amplitude of color changes and is therefore rather sensitive to color gradient artefacts which can be reduced due to the image registration according to the present invention.
• a computer program may be stored/ distributed on a suitable non-transitory medium, such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems.
• a suitable non-transitory medium such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems.

Landscapes

• Health & Medical Sciences (AREA)
• Engineering & Computer Science (AREA)
• General Health & Medical Sciences (AREA)
• Medical Informatics (AREA)
• Public Health (AREA)
• Physics & Mathematics (AREA)
• Epidemiology (AREA)
• Primary Health Care (AREA)
• Biomedical Technology (AREA)
• Life Sciences & Earth Sciences (AREA)
• General Physics & Mathematics (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
• Radiology & Medical Imaging (AREA)
• Theoretical Computer Science (AREA)
• Cardiology (AREA)
• Heart & Thoracic Surgery (AREA)
• Surgery (AREA)
• Business, Economics & Management (AREA)
• Biophysics (AREA)
• Pathology (AREA)
• General Business, Economics & Management (AREA)
• Molecular Biology (AREA)
• Physiology (AREA)
• Animal Behavior & Ethology (AREA)
• Veterinary Medicine (AREA)
• Image Analysis (AREA)
• Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
• Measuring And Recording Apparatus For Diagnosis (AREA)
• Measuring Pulse, Heart Rate, Blood Pressure Or Blood Flow (AREA)
• Image Processing (AREA)

Priority Applications (4)

Applications Claiming Priority (2)

Publications (1)

Family

ID=67296959

Family Applications (1)

Country Status (5)

Families Citing this family (4)

Citations (1)

Family Cites Families (4)

• 2019
  • 2019-07-02 EP EP19183802.8A patent/EP3761317A1/en not_active Withdrawn
• 2020
  • 2020-06-30 US US17/620,189 patent/US20220343517A1/en not_active Abandoned
  • 2020-06-30 EP EP20736300.3A patent/EP3994700A1/en not_active Withdrawn
  • 2020-06-30 WO PCT/EP2020/068337 patent/WO2021001340A1/en not_active Ceased
  • 2020-06-30 CN CN202080054841.4A patent/CN114175096A/zh active Pending
  • 2020-06-30 JP JP2021575936A patent/JP2022537764A/ja not_active Withdrawn

Patent Citations (1)

Non-Patent Citations (11)

Also Published As

Similar Documents

Legal Events