DE102017118636B3 - Method for contactless determination of the heart rate of a person - Google Patents

Abstract

The invention relates to a method for contactless determination of the heart rate (17) of a person using a continuous RGB image sequence, comprising the following steps: a. Recording an RGB image sequence and transmitting the RGB image sequence to a processing unit, b. Selecting at least three regions in a face of a person whose orientation was detected in the RGB image sequence, c. Tracking of the face parallel to step b, wherein each of the suitable pixels is compared in intensity with the intensity of the respective neighboring pixels and both the displacement of the path length of each pixel by a certain height and width from a time step to the temporally following time step, as well evaluated at the temporally preceding time step, d. Selecting appropriate pixels in the selected regions based on facial skin color and landmarks while maintaining a distance of at least two pixels from a skin pixel to a non-skin pixel, e. Signal extraction by transferring a spatial image signal from the RGB image sequence in the selected regions into a time signal, f. Signal processing and evaluation of the time signal, wherein the time signal forms a basis for the determination of the heart rate.

Description

The invention relates to a method for contactless determination of the heart rate of a person and is used in particular for the determination and monitoring of general vitality.

Assistance systems are playing an increasingly important role in a constantly aging society. In recent years, some methods have been developed for this, which can detect the emergency detection optically with the help of cameras. However, these systems can only respond in an emergency. A better approach is to continuously monitor a person's state of health to be proactive before an emergency occurs. In order to accomplish this, a recognition of vital signs is required. The heart rate or heart rate is one of these vital signs. It is possible to determine the heart rate via optical contactless procedures and draw conclusions about the state of health. Conventional procedures such as the electrocardiogram or the pulse oximeter require body contact. However, this is difficult for certain groups of people, e.g. because of dementia or sensitive skin in the elderly or newborn. The contactless determination of the heart rate can thus create an additional advantage.

In addition to Ambient Assisted Living, this technique can be used in e-rehabilitation to record vital signs during a training exercise. In addition, it can be used to detect sudden infant death, to monitor a driver or to triage in the hospital.

The basic method for contactless acquisition of the heart rate is to measure the volume change caused by blood flow. This method is also called plethysmography. The basic contact-based method for photoplethsymography was developed by Hertzman and Spealman in 1937 [1]. It is the basis for the pulse oximeter today. Thus, light changes which tissue radiates its wavelength and its intensity. The pulse can be detected as a change in the intensity of the waveform. In 2008, Verkruysse et al. presented a method that can detect the human heart rate contactless in visible light [2]. They captured videos of people's faces with an RGB camera, set a Region of Interest (ROI) within the face, and spatially averaged all the pixels in the ROI for the three color channels. The frequency was then determined using a Fast Fourier Transform (FFT). This procedure was not yet automated. This methodology was developed by Poh et al. taken up, automated and extended by an Independent Component Analysis (ICA) for greater robustness [3]. Van Gastel et al. have suggested using only the pixels on the forehead instead of the whole face [4]. This is not so much affected by artifacts and has a high signal quality. Instead of ICA, Lewandowska et al. to use a Principal Component Analysis (PCA) [5]. De Haan and Jeanne use a difference signal to reduce reflections [6].

An alternative to these intensity-based methods is the motion-based method of Balakrishnian [7]. It determines the smallest movements of the head, which are caused by the pumping of the blood. The basis is the third Newtonian axiom. Here, methods of optical flow are used.

The EP 2 846 550 A1 describes how a PPG signal can be encoded or decoded. Signal processing does not take place.

In the US 2016/0 015 308 A1 a general system for the determination of vital parameters is described. It has an image acquisition unit and should monitor the vital signs on fitness equipment. It contains filters and a frequency determination. The compensation of movement is described inadequately.

In the US 2016/0306779 A1 A method for determining the heart rate is described in the video. Different methods of windowing and filtering are used. All processing takes place in the time domain. There is no balance of artifacts.

Another method for determining the pulse from video sequences is in the US 2016/0 343 135 A1 disclosed. Image segments are selected and the mean value is determined for these image segments. A principal component analysis (PCA) is used to improve the signal. However, using whole regions instead of individual pixels introduces some inaccuracy. There is also no compensation of artifacts.

The previous methods are suitable for laboratory conditions, but not for practical use. So people usually sit very close to the camera and additional light sources ensure a good illumination of the scene. The main problem is the lack of robustness to artefacts that occur, which significantly reduces accuracy.

In particular, there are two main artifacts that adversely affect robustness. These are movements on the one hand and on the other hand Changes in light intensity. For example, motion artifacts arise when the person moves during the measurement. This is the case in practical applications. Therefore, it is necessary to balance these movements. Intensity artifacts arise when the intensity in the image changes, and in particular in the observed region. This can be caused by shadows and reflections.

The object of the invention is therefore to provide a method for contactless determination of the heart rate of a person available, with the aid of which the heart rate can be robustly detected and in which motion artifacts and intensity artifacts can be minimized or eliminated.

This object is achieved with the characterizing features of the first claim.

Advantageous embodiments emerge from the subclaims.

1. a. Aufnahme einer RGB-Bildsequenz und Übermittlung der RGB-Bildsequenz an eine Verarbeitungseinheit,
2. b. Auswahl von wenigstens drei Regionen in einem in der RGB-Bildsequenz detektierten Gesicht einer Person, dessen Orientierung ermittelt wurde,
3. c. Tracking des Gesichtes parallel zu Schritte b, wobei jedes der geeigneten Pixel in seiner Intensität mit der Intensität der jeweiligen Nachbarpixel verglichen wird und sowohl die Verschiebung der Weglänge eines jeden Pixels um eine bestimmt Höhe und Breite von einem Zeitschritt zum zeitlich darauf folgenden Zeitschritt, als auch zum zeitlich vorangegangenen Zeitschritt ausgewertet
4. d. Auswahl geeigneter Pixel in den ausgewählten Regionen auf Basis der Gesichtshautfarbe und von Landmarken, unter Einhaltung eines Abstandes von wenigstens zwei Pixeln von einem Hautpixel zu einem Nicht-Hautpixel,
5. e. Signalextraktion durch Überführung eines räumlichen Bildsignals aus der RGB-Bildsequenz in den ausgewählten Regionen in ein Zeitsignal,
6. f. Signalverarbeitung und Auswertung des Zeitsignals, wobei das Zeitsignal eine Basis für die Bestimmung der Herzfrequenz bildet.

According to the invention, a method for contactless determination of the heart rate of a person using a continuous RGB image sequence, comprising the following steps:

1. a. Recording an RGB image sequence and transmitting the RGB image sequence to a processing unit,
2. b. Selection of at least three regions in a face detected in the RGB image sequence of a person whose orientation has been determined
3. c. Tracking the face parallel to steps b, wherein each of the appropriate pixels is compared in intensity with the intensity of the respective neighboring pixels and both the displacement of the path length of each pixel by a certain height and width from a time step to the temporally following time step, as well evaluated at temporally preceding time step
4. d. Selecting appropriate pixels in the selected regions based on facial skin color and landmarks while maintaining a distance of at least two pixels from a skin pixel to a non-skin pixel;
5. e. Signal extraction by transferring a spatial image signal from the RGB image sequence in the selected regions into a time signal,
6. f. Signal processing and evaluation of the time signal, wherein the time signal forms a basis for the determination of the heart rate.

Before the detection of the person's face, a white balance of the RGB image sequence is preferably made in order to compensate for a possibly occurring blue or red shift in the image. Correction factors for the red and blue channels are introduced, which make this possible. A real white is thus also depicted as white.

As a basis for the further processes, a detection of the face on the 2D image data is performed. As a result, the face is marked by a so-called bounding box around the face.

To determine facial orientation, prominent points on the face are localized as landmarks. Advantageously, thereby a face can also be detected when it is rotated. In the process, prominent points are localized in the face and can be uniquely determined. The algorithm for determining landmarks is based on X. Zhu and D. Ramanan [8]. Here exists a model of the landmarks as they would appear in the optimal face. By means of a flexible model with tree structure, this model can be transferred to any faces. This procedure ensures accurate detection of the face with respect to the orientation of the face.

Preferably, at least three, more preferably at least four regions in the face are selected which are further used for the determination of the heart rate. In particular, the location of the at least three, preferably at least four, regions which are used for the determination of the heart rate depends relatively on the position of the determined bounding box. The regions are within the bounding box and meet certain distance criteria to the edges of the bounding box.

These regions are referred to as ROI (Regions of Interest). This is necessary because pixels in the area of hair, eyebrows, eyes, glasses, etc. should not be included in the skin color model and are therefore excluded. For this purpose, four ROIs are particularly preferably determined in order to obtain a variance over different regions. The ROIs are placed on the forehead, the nose and the two cheeks.

In order to select suitable pixels, a determination of the facial skin color on the first image of the RGB image sequence is preferably carried out first. In this case, preferably all pixels which are located in the previously selected ROIs are used for the determination of a so-called skin color model

To determine the facial skin color, a conversion of the RGB color space in the HSV color space, since a linear separation in the RGB color space is not possible. In this a division of the color information does not take place in red, green and blue but in color value (H, hue), color saturation (S, saturation) and the brightness value (V, value). From the respective values for color space and saturation an individual skin color model is created.

An individual skin color model is created for each subject. That means there are no fixed default values for all people. On the one hand, this ensures greater accuracy and, on the other hand, makes it possible for this method to be applicable to people with different skin colors. In addition, regions with shadows or strong reflections can be excluded, which is reflected in a later improved robustness of the algorithm.

To reliably capture the face region or bounding box over time in a video, tracking is necessary. Thus, for example, movements of the head can be compensated and the face is permanently within the defined bounding box.

In a first step suitable image features must be found in the face, which are characteristic and can be found again. For this application, the minimum eigen features are preferably used [9]. These calculate the minimum eigenvalue of image regions and are thus able to detect corners.

For the tracking of the features, preference is given to using an optical flow technique known as the KLT method [10]. The intensities of the feature pixel and its neighboring pixels are considered. From one frame to the next frame, the feature is shifted by a certain path length in height and width. This shift is determined during tracking.

In order to increase the robustness of the tracking, it is also preferable to perform a backward tracking. The characteristics from the current frame to the current frame are predefined and their displacement determined. The bidirectional error between the forward shift and the backward shift must not exceed one pixel. Otherwise, the feature is discarded for further calculation.

Under certain conditions, e.g. rapid movements or changes in the perspective of the camera on the face, many features can be lost over time. This is a great disadvantage, because it no longer stable tracking is possible. In the worst case, all features disappear. Therefore, it is necessary to re-detect the face at certain intervals and to redefine the characteristics. For this embodiment, the following provisions apply: In the event that the face can not be detected, e.g. due to the perspective distortion, a continuous tracking is performed without the detection. If there are too rapid movements in the image, a new detection of the face is necessary in any case, otherwise there are large deviations, which have a negative effect on the robustness.

On the basis of the determined displacement, the parameters of the geometric transformation can be determined. These transformation parameters are then applied to the bounding box of the face. As a result of the tracking, there is a bounding box for all frames that points to the corresponding / same region.

Within the determined bounding box of the tracking, for each frame (each time step), the skin color model is used, which was previously determined in the first frame. The threshold values for color value and saturation are taken and applied to all pixels. As a result, suitable skin pixels are available for determining the heart rate. However, there are smooth transitions at the edges to other objects on the face, which are detrimental. To counteract this, a spatial distance k in pixels is defined, which must lie between a skin pixel and a non-skin pixel. Preferably k is assumed to be at least 2, more preferably at least 9. This selects suitable (reliable) skin pixels from suitable skin pixels.

The result is skin pixels that are not disturbed by shadows, reflections or other intensity-affecting artifacts.

For signal extraction, for each color channel of R, G and B, the sum of all pixel intensities of the previously selected suitable pixels is formed and divided by the number of these pixels.

• - Normalisierung aller drei zuvor ermittelten Kanäle
• - Anwendung eines Bandpassfilters auf alle drei Kanäle
• - Unabhängigkeitsanalyse
• - Auswahl der unabhängigen Komponente mit der höchsten Periodizität unter Berücksichtigung der Standardabweichung jeder unabhängigen Komponente

In order to suppress further artifacts and noise and to increase the signal-to-noise ratio, signal processing is necessary. Preferably, at least the following steps are performed in the signal processing:

• - Normalization of all three previously determined channels
• - Apply a bandpass filter to all three channels
• - Independence analysis
• - Selection of the independent component with the highest periodicity, taking into account the standard deviation of each independent component

In the preferably first step of the signal processing, the three time signals of R, G and B are normalized. This means that the mean value is formed over the time axis and this is then subtracted. Subsequently, the reciprocal of the standard deviation of the channel is multiplied. Through this step, all three channels are centered around the x-axis (zero axis).

In order to limit the search range for later frequency determination, only frequency values that can occur physiologically in humans should be taken into account. For this purpose, a bandpass filter is preferably used, which only lets frequencies between 0.7 Hz and 4 Hz through. In particular, a 200 order FIR filter is used. This is applied to all three channels.

Despite previous efforts, there are still noise sources in the three R, G, and B channels. To further reduce this, an ICA (Independent Component Analysis) is performed. It is assumed that the three channels are only measurements from three independent sources, which have been combined via a so-called mixed matrix. For the three channels, one is the wanted heart rate signal and two are considered noise sources. With the help of the ICA, the mixed matrix is determined and a system of three independent sources is generated.

One problem after using the ICA is that you do not know which of the three channels is the wanted channel with the wanted signal. This is solved with the channel determination.

In general, it can be stated that the human heart rate changes only slightly in short time intervals. Therefore, the independent component with the highest periodicity should be selected. The periodicity is defined as a proportion of the frequencies on the entire power spectrum with a band width of 0.5 Hz maximum around the dominant frequency of the signal.

However, short but intense motion artifacts can also have a high periodicity. To identify these cases, the standard deviation of each independent component is determined. If the difference to one exceeds a certain threshold d, the component is excluded in the further decision process.

As a result of this step, there is only one more channel that most likely contains the heart rate. This channel is considered for further analysis.

Since the heart rate can change over time, it makes sense to break the remaining channel into individual segments. Preferably, each segment has a sliding window of at most 10 seconds in length. The overlap of the segments is preferably at most 90%.

For the actual frequency determination, preferably at least two different methods are used. The FFT (Fast Fourier Transform) and Welch spectral estimation have proven to be particularly suitable. For these two methods, a transformation of the signal from the time domain into the frequency domain (spectrum) takes place. Subsequently, the dominant peak can be assumed as an estimator of the heart rate.

By estimator is meant an estimate of the heart rate.

In the final step, the heart rate estimates should be confirmed or modified. The priority is to avoid leaps in heart rate. Since the segments overlap to a maximum of 90 percent, it can be assumed that the heart rate can not change abruptly. The previous segments therefore serve as the basis for the next segment. Preferably, it is believed that the heart rate will not change more than 15 beats per minute between two segments (in about 0.1 seconds). This corresponds to 0.25 Hz. Within this interval, the dominant peak in the spectrum is determined. It is no longer the dominant peak across the spectrum.

Advantageously, both motion artifacts and intensity artifacts, which usually occur in comparable methods, can be reduced or eliminated by the method according to the invention. This goes along with a more accurate determination of the heart rate and improved robustness. By the method according to the invention, a preventive monitoring of a person can take place in order to avoid the occurrence of an emergency. In addition to Ambient Assisted Living, this technique can be used in e-rehabilitation to record vital signs during a training exercise. In addition, it can be used to detect sudden infant death, to monitor a driver or to triage in the hospital.

Since the method is an optical non-contact method which requires no direct physical contact with the person, it is particularly suitable for groups of people such as people with dementia or sensitive skin, the elderly, newborns or people in need of care in general.

The invention will be explained in more detail with reference to an embodiment and associated drawings, without being limited to these.

• 1: ein Fließschema des erfindungsgemäßen Verfahrens,
• 2: eine schematische Darstellung eines detektierten Gesichtes nach Erstellen der Landmarken und der Bounding-Box,
• 3: eine schematische Darstellung eines detektierten Gesichtes nach Bestimmung der Regions of Interest (ROI),
• 4: Darstellung der räumlich gemittelten Pixel über die Zeit für die drei Farbkanäle Rot, Grün und Blau.

Showing:

• 1 : a flow chart of the process according to the invention,
• 2 : a schematic representation of a detected face after creating the landmarks and the bounding box,
• 3 : a schematic representation of a detected face after determination of the regions of interest (ROI),
• 4 : Representation of spatially averaged pixels over time for the three color channels red, green and blue.

1 shows a flow chart of the method according to the invention. The proposed system consists of many sub-steps. First, a continuous image sequence, referred to as video below, is recorded with the aid of an RGB camera. 1. A face can be recognized in the image sequence. To suppress specifics of the environment and the image sensor setting, a white balance is set 2 performed. Following is a detection 3 of the face instead. This is followed by the determination of the facial orientation 4 , If this is known, regions of interest (ROI) can be selected according to a special procedure. 5. A determination of facial skin color is made based on the first image of the video sequence 6 of the respective subject in front of the camera. In the running video, a determination of image characteristics is made for each image 7 The basis for subsequent tracking 8th are. After tracking, appropriate pixels are selected according to the previously created skin color model 9. These pixels form the basis for the time signal extraction 10 , This is followed by signal normalization 11 , a bandpass filtering 12 and an Independent Component Analysis 13 , Subsequently, the optimal channel of the independent components is determined 14 and a frequency determination 15 carried out. The method is powered by adaptive filtering 16 completed. Finally, the heart rate 17 be determined. It results from the highest peak after the frequency determination.

In 2 is a schematic representation of a detected face after creating the landmarks 3.2 and the bounding box 3.3 , From the face detection ( 3 in 1 ) results in a first bounding box 3.1 marking the face. Subsequently, landmarks 3.2 localized within the face, which mark particularly prominent points. Based on the landmarks 3.2 becomes a second bounding box 3.3 set, which surrounds a narrower range than the first bounding box 3.1 , Here are all landmarks 3.2 within the second bounding box 3.3 and definitely meet distance criteria for limiting the second bounding box 3.3 ,

3 shows a schematic representation of a detected face after determination of the regions of interest (ROI) ( 5 in 1 ). This will be within the previously determined second bounding box 3.3 determined multiple ROIs to obtain a variance across different regions. The ROIs 5.1 . 5.2 . 5.3 be on your forehead 5.1 , the two cheeks 5.2 and the nose 5.3 placed. The localization of the ROIs 5.1 . 5.2 . 5.3 depends relatively on the position in the second bounding box 3.3 ,

A representation of the spatially averaged pixels over time for the three color channels red, green and blue is in 4 shown. By combining the tracking ( 8th in 1 ) and the skin color model ( 6 in 1 ) many motion artifacts and intensity artifacts have already been reduced or eliminated. Subsequently, the spatial image signal is to be converted into a time signal. Each color channel of red R, green G and blue B is considered separately. For each channel, the sum of all pixel intensities of the pixels selected by the color model is formed and divided by the number of these pixels. Applying this frame by frame over time results in a time course, as in 4 seen. This step is called signal extraction ( 10 in 1 ) designated.

quoted non-patent literature:

• [1] Hertzman, AB and Spealman, CR (1937). Observations on the finger volume pulse recorded photoelectrically. American Journal of Physiology, 119: 334-335 ,
• [2] Verkruysse, W., Svaasand, LO, and Nelson, JS (2008). Observations on the finger volume pulse recorded photoelectrically. Optics Express, 16 (26): 21434-21445 ,
• [3] Poh, M.Z., McDuff, D., and Picard, R. (2010). Noncontact, automated cardiac pulse measurements using video imaging and blind source separation. Optics Express, 18 (10): 10762-10774 ,
• [4] van Gastel, M., Zinger, S., Kemps, H., and de With, P. (2014). e-health video system for performance analysis in heart revalidation cycling. In Consumer Electronics Berlin (ICCE-Berlin), 2014 IEEE Fourth International Conference on, pages 31-35 ,
• [5] Lewandowska, M., Ruminski, J., Kocejko, T., and Nowak, J. (2011). Measuring pulse rate with a webcam - a non-contact method for evaluating cardiac activity. In Computer Science and Information Systems (FedCSIS), 2011 Federated Conference on, pages 405-410 ,
• [6] de Haan, G. and Jeanne, V. (2013). Robust pulse rate from chrominance-based rppg. Biomedical Engineering, IEEE Transactions on, 60 (10): 2878-2886 ,
• [7] Balakrishnan, G., Durand, F., and Guttag, J. (2013). Detecting pulse from head motions in video. In Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on, pages 3430-3437 ,
• [8th] X. Zhu and D. Ramanan. Face detection, pose estimation, and landmark localization in the wild. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Patents Recognition, pages 2879-2886
• [9] Shi, J. and Tomasi, C. (1993). Good features to track. Technical report, Cornell University, Ithaca, NY, USA ,
• [10] Tomasi, C. and Kanade, T. (1991). Detection and Tracking of Point Features. Technical report, Carnegie Mellon University.

LIST OF REFERENCE NUMBERS

1

Image capture camera

2

White balance

3

face detection

3.1

first bounding box

3.2

landmarks

3.3

second bounding box

4

Determination of facial orientation

5

Selection of Regions of Interest (ROI)

5.1

ROI on the forehead

5.2

ROIs on the cheeks

5.3

ROI on the nose

6

Determination of facial skin color / a skin color model

7

Determination of image features

8th

face tracking

9

Selection of suitable pixels on the skin

10

Time signal extraction

11

signal normalization

12

Bandpass filtering

13

Independent Component Analysis

14

determining channel

15

frequency determination

16

adaptive filtering

17

heart rate

Claims (10)

A method for contactless determination of a person's heart rate using a continuous RGB image sequence, comprising the following steps: Recording an RGB image sequence (1) and transmitting the RGB image sequence to a processing unit, Selecting at least three regions (5) in a (3) face detected in the RGB image sequence of a person whose orientation has been determined (4), Tracking of the face (8) parallel to step b, wherein each of the appropriate pixels is compared in intensity with the intensity of the respective neighboring pixels and both the displacement of the path length of each pixel by a determined height and width from one time step to the next in time Time step, as well as evaluated at the temporally preceding time step, Selection of suitable pixels (9) in the selected regions based on facial skin color and landmarks, while maintaining a distance of at least two pixels from a skin pixel to a non-skin pixel, Signal extraction (10) by converting a spatial image signal from the RGB image sequence in the selected regions into a time signal, - Signal processing and evaluation of the time signal, wherein the time signal forms a basis for the determination of the heart rate (17). Method according to Claim 1 , characterized in that before the detection (3) of the face of the person, a white balance (2) of the RGB image sequence is made. Method according to Claim 1 or 2 , characterized in that for determining the facial orientation (4) prominent points in the face as landmarks (3.2) are located. Method according to one of Claims 1 to 3 , characterized in that the localization of the at least three regions (5) used for the determination of the heart rate (17) depends relatively on the position of the determined landmarks (3.2) and the regions (5.1, 5.2, 5.3) Do not include landmarks (3.2). Method according to one of Claims 1 to 4 , characterized in that for the selection of suitable pixels (9), first a determination of the facial skin color (6) on the first image of the RGB image sequence takes place. Method according to one of Claims 1 to 5 , characterized in that for the determination of the facial skin color (6), a conversion of the RGB color space in the HSV color space is carried out and from the respective values for color space and saturation, an individual skin color model is created. Method according to one of Claims 1 to 6 , characterized in that for signal extraction (10) for each color channel of R, G and B the sum of all pixel intensities of the previously selected suitable pixels is formed and divided by the number of these pixels. Method according to one of Claims 1 to 7 Characterized in that in the signal processing, the following steps are carried out at least: - Normalization of all three previously identified channels - application of a bandpass filter on all three channels - independence analysis - selection of the independent component with the highest frequency in consideration of the standard deviation of each independent component Method according to one of Claims 1 to 8th , characterized in that in the analysis, the determination of the heart rate by means of fast Fourier transformation and / or a spectral estimation Welch takes place and as a result, an estimator for the heart rate is present. Method according to Claim 9 , characterized in that the estimator for the heart rate is confirmed or modified by adaptive filtering.

Images