KR20210145946A - Device and method for camera based photoplethysmogram estimation using chrominance - Google Patents

Abstract

The present invention relates to a device and method for estimating a non-contact photoplethysmography. According to one embodiment of the present invention, the device and method for estimating the non-contact photoplethysmography enable to estimate the photoplethysmography robust to movement with the camera-based non-contact biometric information. The device for estimating the non-contact photoplethysmography comprises: an input part; an area of interest part; a filtering part; a conversion part; and an extracting part.

Description

Apparatus and method for non-contact photoplethysmography estimation {DEVICE AND METHOD FOR CAMERA BASED PHOTOPLETHYSMOGRAM ESTIMATION USING CHROMINANCE}

The present invention relates to a non-contact PPL estimation apparatus and method, and more particularly, to a non-contact PPL estimation apparatus and method for recognizing a biosignal by estimating PPL in a non-contact manner using a color difference of a camera-based image.

The bio-signals obtained from the human body can be used as indicators that reflect an individual's health and psychological state in real time. In order to measure biosignals, contact-based electrocardiography (ECG) and photoplethysmography (PPG) methods are usually used. Since these contact measurement methods are measured by directly attaching electrodes or sensors to the user's body, they cannot be applied when the user feels uncomfortable in hygiene or has sensitive skin.

Background art of the present invention is published in Korean Patent Laid-Open No. 10-2018-0131262.

The present invention provides an apparatus and method for estimating a vision-based non-contact photoplethysmogram that is robust to motion using a color difference signal.

The present invention provides an apparatus and method for estimating a non-contact photoplethysmogram using biometric information that can be acquired through camera-based non-contact.

The present invention provides an apparatus and method for estimating a heart rate by estimating a photoplethysmogram using non-contact biometric information.

According to one aspect of the present invention, there is provided a non-contact photoplethysmography wave estimation apparatus.

According to an embodiment of the present invention, an input unit for acquiring a plurality of images, a region of interest unit for setting a region of interest in the image, a filtering unit for filtering the region of interest, and converting pixels of the filtered region of interest into a color difference signal It may include a conversion unit and an extractor for estimating a photoplethysmogram waveform from the color difference signal.

According to another aspect of the present invention, there is provided a non-contact photoplethysmography method and a computer-readable recording medium in which a computer program executing the method is recorded.

A method for estimating a non-contact photoplethysmogram and a recording medium storing a computer program executing the method according to an embodiment of the present invention include: acquiring a plurality of images; setting a region of interest in the image; and filtering the region of interest , converting the filtered pixels of the region of interest into a color difference signal and estimating a photoplethysmogram waveform from the color difference signal.

According to an embodiment of the present invention, the apparatus and method for estimating the photoplethysmogram may estimate the photoplethysmogram using camera-based non-contact biometric information.

According to an embodiment of the present invention, the non-contact PPL estimation apparatus and method can estimate the PPV robust to the movement of a measurement object when camera-based biometric information is acquired.

1 to 5 are diagrams for explaining an apparatus for estimating a non-contact photoplethysmogram according to an embodiment of the present invention.
6 and 7 are diagrams illustrating a method for estimating a non-contact PPV according to an embodiment of the present invention.
8 and 9 are exemplary screens of photoplethysmography waveform results and simultaneously measured reference signals for a face video without motion and a face video with motion.
10 is an example of an experiment measuring a signal-to-noise ratio (SNR) when extracting a photoplethysmography waveform;

Since the present invention can have various changes and can have various embodiments, specific embodiments are illustrated in the drawings and will be described in detail through detailed description. However, this is not intended to limit the present invention to specific embodiments, and it should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present invention. In describing the present invention, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. Also, the expressions "a" and "a", "a" and "a", as used in this specification and claims, should generally be construed to mean "one or more" unless stated otherwise.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. do it with

1 to 5 are diagrams for explaining an apparatus for estimating a non-contact photoplethysmogram according to an embodiment of the present invention.

Referring to FIG. 1 , the non-contact PPG wave estimation apparatus 10 may estimate the LPT wave in a non-contact manner by using a color difference signal of a face image obtained based on an RGB camera. Since the non-contact photoplethysmogram estimation apparatus 10 estimates the photoplethysmogram in a non-contact manner, it is hygienic even in an environment used by a large number of users and is robust against movement by using a color difference signal.

Referring to FIG. 2 , the apparatus 10 for estimating a non-contact photoplethysmogram includes an input unit 100 , a region of interest unit 200 , a filtering unit 300 , a converting unit 400 , an extracting unit 500 , and a determining unit 600 . ) is included.

The input unit 100 acquires a plurality of images through a camera. For example, the input unit 100 acquires a time-series image through an RGB camera. The input unit 100 expresses the plurality of acquired images as pixels.

3 is an example in which a plurality of images acquired by the input unit 100 are expressed as pixels. For example, the input unit 100 includes a plurality of time-series images.

Referring back to FIG. 2 , the region of interest unit 200 sets a specific target region for estimating the plethysmographic wave in an image represented by pixels as a region of interest (ROI). For example, the region of interest unit 200 may set the face region as a region of interest (ROI).

4 is an example in which the region of interest set by the region of interest unit 200 is indicated by a thick dotted line. The region of interest unit 200 may use a region tracking algorithm to set a consistent region of interest (ROI) from a plurality of images connected in time series. More specifically, the region of interest unit 200 may obtain a consistent region of interest (ROI) by continuously tracking the position of the target based on the initially registered target region information.

Referring back to FIG. 2 , the filtering unit 300 filters the skin pixels within the ROI. For example, the region of interest (ROI) may include pixels related to hair, eyebrows, or background that are less relevant to the plethysmography. The filtering unit 200 may use skin color filtering based on the spatial and color information to exclude a region not related to the plethysmographic wave estimation.

FIG. 5 is an example in which the filtering unit 300 detects pixels corresponding to an area not related to the PPV through skin color filtering and displays the pixels as black pixels.

Referring back to FIG. 2 , the conversion unit 400 converts the skin pixel of the ROI of FIG. 5 into a color difference signal. For example, the converter 400 may use the YCbCr color space to perform color difference signal conversion. YCbCr has a characteristic of expressing a luminance component (Y) and a color difference component (Cb, Cr) separately. When estimating the PPL, uneven illumination reflection on the skin surface caused by large and small movements of the measurement target area may act as noise (noise) on the PPL waveform. Non-uniform light reflections on the skin surface are mainly related to the luminance of the image. The conversion unit 400 may estimate the photoplethysmography waveform by using the color difference component except for the luminance component.

The extraction unit 500 extracts a waveform of the PPV by using the color difference component of the image. For example, the extractor 500 may use an average of color difference signals of all skin pixels as a representative signal value. Alternatively, the extractor 500 may use a preselected fixed number of skin pixels as a signal value. The extractor 500 may calculate a time-series PPV waveform based on signal values obtained from a plurality of images.

The extractor 500 may filter low-frequency and high-frequency components that are not related to heartbeat activity from the photoplethysmography waveform by applying an additional bandpass filter. For example, the bandpass filter may use a passband of 0.7 to 3.0 Hz corresponding to 42 to 180 beats per minute (bpm).

The determination unit 600 estimates an average heart rate from the filtered photoplethysmography waveform.

The determination unit 600 calculates the power spectral density of the photoplethysmography waveform. For example, the determination unit 600 calculates a spectral expression through a Fourier transform on the time series signal of the photoplethysmogram waveform. The determination unit 600 calculates the spectral density by integrating the spectral power in the spectral representation of the photoplethysmography waveform.

The determination unit 600 determines a representative frequency corresponding to the global maximum value of the power spectral density. The determination unit 600 may determine one frequency having the greatest power as the representative frequency by comparing the power levels of each frequency. Since the purpose of the determination unit 600 is to find a frequency corresponding to the average heart rate, an effective band is set for a band related to the heart rate, and power magnitude comparison is performed. For example, the determination unit 600 may use a band of 0.7 to 3.0 Hz corresponding to 42 to 180 bpm as an effective band.

The determiner 600 may convert the determined representative frequency into a desired unit of time to estimate the average heart rate for the measurement section.

6 and 7 are diagrams illustrating a method for estimating a non-contact photoplethysmogram according to an embodiment of the present invention. Each of the processes to be described below is a process performed by each functional unit constituting the non-contact photoplethysmography apparatus, but for the sake of concise and clear explanation of the present invention, the subject of each step will be collectively referred to as a non-contact photoplethysmography apparatus.

Referring to FIG. 6 , in step S610 , the non-contact photoplethysmography apparatus 10 acquires a plurality of images through a camera. For example, the non-contact photoplethysmography apparatus 10 may acquire a time-series image including the skin through an RGB camera.

In step S620 , the non-contact PPL estimation apparatus 10 sets a specific target area to which the PPL measurement is to be applied as a region of interest. For example, the non-contact photoplethysmography apparatus 10 may set the face region as the region of interest. The non-contact photoplethysmography apparatus 10 may include a region tracking algorithm to set a consistent region of interest according to time in a plurality of images.

In operation S630 , the non-contact photoplethysmogram estimating apparatus 10 filters the skin pixels in the ROI. The non-contact PPL estimation apparatus 10 may remove pixels that are not related to the PPL included in the region of interest. The non-contact PPL estimation apparatus 10 may detect and remove pixels not related to the PPL estimation by using skin color filtering based on spatial and color information. For example, the non-contact photoplethysmography apparatus 10 may detect pixels corresponding to hair, eyebrows, background, etc. that are not related to the estimation of the photoplethysmography in the region of interest, display them as black pixels, and exclude irrelevant components. .

In operation S640 , the non-contact photoplethysmogram estimator 10 converts the skin pixel into a color difference signal. The non-contact photoplethysmogram estimator 10 may convert a color difference signal using the YCbCr color space. YCbCr can be expressed by separating the luminance component (Y) and the chrominance component (Cb, Cr).

In operation S650 , the non-contact PLW estimator 10 may estimate the PL waveform by discarding the luminance Y that may cause noise and using the chrominance components Cb and Cr. Non-uniform light reflections on the skin surface caused by large and small movements of the target area during PPV measurement may act as noise (noise) when extracting PPL waveforms. Since the non-uniform light reflection on the skin surface is mainly related to the luminance component, the non-contact photoplethysmography apparatus 10 discards the luminance Y and uses the chrominance components (Cb, Cr) to remove the noise. to estimate

The non-contact photoplethysmography apparatus 10 may estimate a waveform of a time-series photoplethysmogram based on signal values obtained from a plurality of images. For example, the non-contact photoplethysmography apparatus 10 may use an average of color difference signals of all skin pixels as a representative signal value. Alternatively, the non-contact photoplethysmogram estimator 10 may use a preselected fixed number of skin pixels as a representative signal value.

Referring to FIG. 7 , the non-contact photoplethysmogram estimator 10 estimates an average heart rate from the photoplethysmography wave waveform.

In step S710 , the non-contact PPL estimating apparatus 10 filters low-frequency and high-frequency components not related to heartbeat activity in the PPL waveform by applying an additional bandpass filter. For example, the non-contact photoplethysmography apparatus 10 may use a band-pass filter of 0.7 to 3.0 Hz corresponding to 42 to 180 beats per minute (bpm).

In step S720 , the non-contact PPL estimation apparatus 10 calculates the power spectral density of the PPL waveform that has undergone band-pass filtering. For example, the non-contact photoplethysmography apparatus 10 calculates a spectral representation through a Fourier transform on a time series signal of the photoplethysmographic wave waveform. The non-contact PPL estimator 10 calculates a spectral density by integrating the spectral power in the spectral representation of the PPL waveform.

In step S730 , the non-contact photoplethysmography apparatus 10 calculates a frequency corresponding to the maximum power value of the power spectral density. The non-contact photoplethysmogram estimator 10 determines one representative frequency having the greatest power by comparing the power magnitudes of each frequency. Since the purpose of the non-contact photoplethysmogram estimator 10 is to find a frequency corresponding to the average heart rate, a band related to the heart rate is set as an effective band to compare power levels. For example, the non-contact photoplethysmography apparatus 10 may use a band of 0.7 to 3.0 Hz corresponding to 42 to 180 bpm as an effective band. The effective heart rate range is not a fixed value, but can be changed according to the purpose of use. For example, an effective heart rate range of 42-240 bpm can be used as wide, or a narrower interval can be set as needed.

In step S740 , the non-contact plethysmographic wave estimator 10 converts a representative frequency determined by comparing power levels into a desired unit of time to estimate an average heart rate for the measurement section.

8 and 9 are exemplary screens of a motion picture of a face without motion, a photoplethysmography wave waveform result of a motion picture of a face with motion, and a reference signal measured at the same time.

FIG. 8 is an example illustrating PPV waveform results estimated with red, green, blue, Y, and Cb+Cr channel signals from a motionless face video and a reference signal measured at the same time.

FIG. 9 is an example illustrating PPV waveform results estimated with red, green, blue, Y, and Cb+Cr channel signals from a facial video in which a movement (yaw) is present, and a reference signal measured at the same time. When there is movement, the non-uniform illumination change phenomenon occurring on the skin surface is conspicuous, so the PPV waveforms of most channels are polluted.

Referring to FIGS. 8 and 9 , a periodic heartbeat waveform is observed in the Cb+Cr channel signal, whereas the motion component dominates the photoplethysmography waveform in most channel signals. Accordingly, the non-contact PPG wave estimator 10 extracts a PPL waveform by using a color difference signal excluding the luminance for estimating the heart rate.

10 is an experimental example of measuring a signal-to-noise ratio (SNR) when extracting a photoplethysmography waveform.

For the experiment, a 30-second long RGB image recorded at 30 fps with a resolution of 640 X 480 pixels was analyzed, and the corresponding photoplethysmogram signal of the finger was simultaneously measured and compared with Ubpulse 360. The average heart rate was calculated as the band with the maximum power at 42-240 bpm of the power spectral density for the pulsed signal.

Referring to FIG. 10 , there is no problem in the face image without motion, but in the case of the face rotation image, the signal-to-noise ratio (SNR) is the best performance of the photoplethysmography waveform estimation using the Cb+Cr signal compared to other signals. can be checked

Table 1 is an example of an experiment comparing the Root-Mean-Squaur Error (RMSE) of each signal channel in the same experiment.

Channel
＼
Scenario G Y Cb+Cr L* a*+b* stable 1.16 1.73 1.20 1.82 5.92 yaw 12.26 12.25 6.29 12.28 10.79 pitch 16.96 17.05 10.58 17.09 9.81 roll 11.04 12.44 2.81 12.42 10.92

The above-described non-contact photoplethysmography method may be implemented as a computer-readable code on a computer-readable medium. The computer-readable recording medium may be, for example, a removable recording medium (CD, DVD, Blu-ray disk, USB storage device, removable hard disk) or a fixed recording medium (ROM, RAM, computer-equipped hard disk). can The computer program recorded on the computer-readable recording medium may be transmitted to another computing device through a network such as the Internet and installed in the other computing device, thereby being used in the other computing device.

In the above, even though it has been described that all components constituting the embodiment of the present invention are combined or operated as one, the present invention is not necessarily limited to this embodiment. That is, within the scope of the object of the present invention, all the components may operate by selectively combining one or more.

Although acts are shown in a particular order in the drawings, it should not be understood that the acts must be performed in the specific order or sequential order shown, or that all illustrated acts must be performed to obtain a desired result. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of the various components in the embodiments described above should not be construed as necessarily requiring such separation, and the described program components and systems may generally be integrated together into a single software product or packaged into multiple software products. It should be understood that there is

So far, the present invention has been focused on the embodiments thereof. Those of ordinary skill in the art to which the present invention pertains will understand that the present invention can be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the present invention is indicated in the claims rather than the foregoing description, and all differences within the scope equivalent thereto should be construed as being included in the present invention.

10: non-contact photoplethysmogram estimation device
100: input unit
200: region of interest
300: filtering unit
400: conversion unit
500: extraction unit
600: judgment unit

Claims (11)

A non-contact photoplethysmogram estimation apparatus comprising:
an input unit for acquiring a plurality of images;
a region of interest unit configured to set a region of interest in the image;
a filtering unit filtering the region of interest;
a conversion unit converting the filtered pixels of the region of interest into a color difference signal; and
and an extractor for estimating a photoplethysmography waveform from the color difference signal.
According to claim 1,
and a determination unit for estimating an average heart rate from the photoplethysmography waveform.
According to claim 1,
A filtering unit for filtering the region of interest
A non-contact photoplethysmogram estimation device that displays pixels that are not related to skin as black pixels through skin pixel filtering.
2. The method of claim 1
a conversion unit that converts the filtered pixels of the region of interest into a color difference signal
A non-contact photoplethysmography apparatus for separating only a color difference signal excluding a luminance component from among skin pixels in the region of interest.
According to claim 1,
An extracting unit for estimating a photoplethysmogram waveform from the color difference signal
A non-contact photoplethysmography wave estimator for calculating a time-series photoplethysmogram waveform from the color difference signal values obtained from the plurality of images.
A non-contact photoplethysmography method comprising:
acquiring a plurality of images;
setting a region of interest in the image;
filtering the region of interest;
converting the filtered pixels of the region of interest into a color difference signal; and
and estimating a PPV waveform from the color difference signal.
7. The method of claim 6,
The method further comprising the step of estimating an average heart rate from the photoplethysmography waveform.
7. The method of claim 6,
The step of filtering the region of interest
A non-contact photoplethysmography method in which pixels not related to skin are displayed as black pixels through skin pixel filtering.
7. The method of claim 6
The step of converting the filtered pixel of the region of interest into a color difference signal
A non-contact photoplethysmography method for separating only a color difference signal excluding a luminance component from among skin pixels in the region of interest.
7. The method of claim 6,
The step of estimating the photoplethysmography waveform from the color difference signal comprises:
A method for estimating a non-contact PPL for calculating a time-series PPL waveform using the color difference signal values obtained from the plurality of images.
A computer program recorded on a computer-readable recording medium for executing any one of the non-contact photoplethysmographic wave estimation methods of claims 6 to 10.

Images