WO2019102535A1 - Pulse wave detection device, pulse wave detection method, and storage medium - Google Patents

Abstract

In order to suppress reduction in the accuracy of pulse wave information, caused by movement of the face of a measurement subject during pulse wave detection, this pulse wave detection device comprises: a region-of-interest setting unit that sets a region-of-interest for the face of a measurement subject, in chronological images that capture the face of the measurement subject; a brightness-data generation unit that generates brightness data being chronological data for representative values of brightness values for a prescribed color component, for pixels in the region-of-interest; a displacement data generation unit that generates, on the basis of the chronological images, displacement data indicating the movement of the face of the measurement subject; a pulse wave component extraction unit that uses the displacement data and extracts a signal component caused by pulse waves, in the brightness data; and an output unit that outputs pulse wave information on the basis of the extracted signal component.

Description

Pulse wave detection device, pulse wave detection method, and storage medium

The present disclosure relates to a technique for acquiring pulse wave information from an image.

There is known a method of photographing a face of a subject and detecting a pulse wave from a moving image obtained by photographing. For example, Non-Patent Document 1 and Patent Documents 1 to 3 are documents describing a method of detecting a pulse wave from a moving image of a face. The “pulse wave” is a change in volume of blood in a blood vessel. Analysis of the pulse wave can, for example, identify the pulse rate (which is usually the same as the heart rate) which is one piece of pulse wave information.

Non-Patent Document 1 performs independent component analysis of temporal change of each average value of luminance of R component (Red component), G component (Green component), and B component (Blue component) detected from a face area The method of estimating the pulse rate after processing by (ICA: Independent Component Analysis) is described. In this method, after converting the independent component into a frequency spectrum, the pulse rate is estimated from the frequency of the peak included in the frequency spectrum.

Patent Document 1 also describes a technique for detecting a pulse rate from a spectral distribution obtained by independent component analysis.

By the way, when the subject moves during shooting, if the area where the luminance value is detected in the image (also referred to as a measurement area or a region of interest) is fixed, the part of the face included in the measurement area changes. The detected luminance value changes. That is, changes in the position and orientation of the face cause noise to the signal indicating the pulse wave. Therefore, in order to obtain pulse wave information more accurately, it is important to take measures against noise caused by changes in the position and orientation of the face.

Patent Document 2 discloses a pulse wave detection device that performs processing for preventing a decrease in accuracy even when there is a lateral movement of a face by observing detection accuracy at the boundary of a measurement region.

Patent Document 3 measures the position of a “sample” (a pair of small regions consisting of a plurality of pixels) for acquiring a pulse wave signal, and calculates the pulse rate from the signal of a sample determined to be reliable. An apparatus is disclosed.

Patent Document 4 describes a pulse wave detection device that detects the rotation angle of the face direction of the person to be measured with respect to a predetermined direction, and calculates the position of the measurement region of the pulse wave based on the detected rotation angle. ing.

JP 2012-239661 A JP, 2014-198200, A JP, 2017-93760, A JP, 2014-198201, A

Even if the measurement area is set to interlock with the range of the specific part of the face by the techniques described in Patent Documents 2 to 4, the detection is performed by the change in the direction of the specific part or the change in the light hit. The luminance values extracted from the area include noise due to the change in the orientation of the face.

An object of the present invention is to provide an apparatus, a method, and the like that suppress the decrease in the accuracy of pulse wave information due to the movement of the face of a subject.

A pulse wave detection device according to an embodiment of the present invention is a region-of-interest setting means for setting a region of interest of the face of the subject with respect to a time-series image of the face of the subject Luminance data generating means for generating luminance data which is time-series data of representative values of luminance values of predetermined color components of pixels in the pixel, and displacement indicating the movement of the face of the subject based on the time-series image Pulse wave component extraction means for extracting signal components attributable to pulse waves in the luminance data using displacement data generation means for generating data, and the displacement data, Pulse wave based on the extracted signal elements And output means for outputting the information of

In the pulse wave detection method according to one embodiment of the present invention, a region of interest of the face of the subject is set in a time-series image in which the face of the subject is photographed, and Luminance data, which is time-series data of representative values of luminance values of predetermined color components, is generated, displacement data indicating the movement of the face of the subject is generated based on the time-series image, and the displacement data is used It extracts a signal component attributable to a pulse wave in the luminance data, and outputs pulse wave information based on the extracted signal component.

A program according to an embodiment of the present invention includes a computer, a region of interest setting process for setting a region of interest of the face of the subject for a time-series image in which the face of the subject is photographed, A luminance data generation process of generating luminance data which is time-series data of representative values of luminance values of predetermined color components of pixels in the pixel, and displacement indicating the movement of the face of the subject based on the time-series image A displacement data generation process for generating data, a pulse wave component extraction process for extracting a signal component attributable to a pulse wave in the luminance data using the displacement data, and a pulse wave based on the extracted signal component And an output process of outputting information of The program is stored in, for example, a computer readable non-transitory storage medium.

ADVANTAGE OF THE INVENTION According to this invention, the fall of the precision of the information of the pulse wave by moving a to-be-measured person's face is suppressed.

It is a block diagram showing composition of a pulse wave detecting device concerning a 1st embodiment of the present invention. It is a figure which shows the example of luminance data. It is a figure which shows the example of displacement data. It is a figure which shows notionally the example of the method of producing | generating correction data from luminance data. It is a figure which shows notionally the method of performing a pulse estimation using independent component analysis. It is a flowchart which shows the flow of operation | movement of the pulse wave detection apparatus which concerns on 1st Embodiment. It is a block diagram which shows the structure of the modification of the pulse wave detection apparatus which concerns on 1st Embodiment. It is a block diagram which shows the structure of the pulse-wave detection apparatus based on the 2nd Embodiment of this invention. It is a block diagram which shows the structure of the pulse-wave detection apparatus based on the 3rd Embodiment of this invention. It is a figure for demonstrating the example of the estimation method of the heart rate which concerns on 3rd Embodiment. It is a block diagram showing composition of a pulse wave detection device concerning one embodiment of the present invention. It is a flowchart which shows the flow of operation | movement of the pulse wave detection apparatus which concerns on one Embodiment. It is a block diagram which shows the example of the hardware which comprises each part of each embodiment of this invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<< First Embodiment >>
First, a first embodiment of the present invention will be described. The pulse wave detection device 11 according to the first embodiment is a device that detects a pulse wave. Specifically, to detect a pulse wave is to find a signal that changes due to the pulse wave from data including a signal that changes due to the pulse wave, and derive information on the pulse wave. Examples of pulse wave information include time-series data of a pulse wave-changed signal and a pulse rate. Hereinafter, detection of a pulse wave is also referred to as pulse wave detection.

[Constitution]
FIG. 1 is a block diagram showing the configuration of the pulse wave detection device 11. In the first embodiment, the pulse wave detection device 11 and the imaging device 30 are communicably connected. The communication method between the pulse wave detection device 11 and the imaging device 30 may be any communication method (for example, communication via a cable or wireless communication).

The pulse wave detection device 11 includes a storage unit 110, a region of interest setting unit 111, a luminance data generation unit 112, a displacement data generation unit 113, a pulse wave information derivation unit 114, and an output unit 119.

The region of interest setting unit 111, the luminance data generation unit 112, the displacement data generation unit 113, the pulse wave information derivation unit 114, and the output unit 119 are realized by, for example, one or more CPUs (Central Processing Units) that execute a program. Be done.

<Imaging Device 30>
The imaging device 30 is an imaging device mounted with an imaging element such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) image sensor. For example, the imaging device 30 mounts a plurality of light receiving elements that measure the intensity of light such as an R (Red) component, a G (Green) component, and a B (Blue) component. The imaging device 30 may be a camera integrally mounted on the pulse wave detection device 11, such as an in-camera and an out-camera mounted on an existing mobile terminal device. Alternatively, the imaging device 30 may be a digital camera, a web camera or the like connected to the pulse wave detection device 11 via an external terminal.

The imaging device 30 captures a moving image of the face of a person to be detected as a pulse wave. The imaging device 30 acquires a moving image by continuously imaging a target person at a predetermined frame rate for a predetermined time (for example, several seconds or several minutes). Hereinafter, data of a plurality of temporally continuous face images (in other words, time-series images) obtained by imaging will be referred to as moving image data.

Each frame of moving image data is, for example, a rectangle of 640 horizontal pixels × 480 vertical pixels. Each pixel included in the frame has color luminance information. The information of the luminance is given by the gradation value. For example, if the moving image is a grayscale moving image, the pixel has a tone value representing luminance. When the moving image is a color moving image, the pixel has, for example, gradation values of respective luminances of R, G, and B components which are three primary colors. The gradation value is represented, for example, by 8-bit information. Note that the gradation value of the pixel is not based on the RGB color system, but is based on another color system (HSV color system, YUV color system, etc.) that can be converted to the RGB value. It may be represented by a value.

In the following description, the gradation value of the luminance of each color given to the pixel is also referred to as "pixel value" or "luminance value".

As a modification, the imaging device 30 may be replaced with a device other than the imaging device that outputs moving image data to the pulse wave detection device 11. Such a device is, for example, a reader that reads moving image data from a storage medium that stores moving image data, or a storage device. In addition, as another modification, the pulse wave detection device 11 may read moving image data from a storage medium such as a flash memory. Alternatively, the pulse wave detection device 11 may read moving image data stored in a storage area in the pulse wave detection device 11.

<Configuration of Each Part of Pulse Wave Detection Device 11>
Hereinafter, a temporal range of moving image data to be processed by each unit of the pulse wave detection device 11 will be referred to as “target data range”. The target data range is the length of part or all of the acquired moving image data. For example, the pulse wave detection device 11 sets 30 seconds as the target data range. If the target data range is shorter than the moving image data, the pulse wave detection device 11 may execute processing relating to the pulse wave detection for each target data range, with a plurality of ranges as different target data ranges. . Note that the plurality of ranges in this case may be ranges in which parts partially overlap with each other.

<Storage unit 110>
The storage unit 110 stores information used by the pulse wave detection device 11 and information generated by processing by the pulse wave detection device 11. The storage unit 110 is, for example, a memory. The storage unit 110 may be an auxiliary storage device such as a hard disk.

The storage unit 110 may store, for example, moving image data output from the imaging device 30.

<Output unit 119>
The output unit 119 outputs the information generated by the pulse wave detection device 11. Examples of output destinations of the output by the output unit 119 include a display device, a storage device, and a communication network. When the output unit 119 outputs information to the display device, the output unit 119 may convert the information so that the display device can display the information. The display device and the storage device described above may be devices outside the pulse wave detection device 11 or may be components included in the pulse wave detection device 11.

<Region of interest setting unit 111>
The region of interest setting unit 111 sets a region of interest. The region of interest in the present disclosure is a region in an image to be subjected to image processing by the pulse wave detection device 11. The region of interest is also referred to as ROI (Region Of Interest). The pulse wave detection device 11 uses the pixel values of the pixels included in the region of interest to detect pulse wave information.

The region of interest setting unit 111 may set the region of interest, for example, according to a predetermined algorithm (that is, automatically). For example, the region of interest setting unit 111 may detect a feature point of a face in a representative frame, and set the region of interest in accordance with a predetermined algorithm based on the coordinates of the feature point.

An existing face feature point detection method may be used to detect the face feature points. As a face feature point detection method, for example, a detection method combining detection of Haar-like features and learning by AdaBoost (Adaptive Boosting) may be mentioned.

The predetermined algorithm may be any algorithm as long as it uses a feature point to set a region. As an example, the region-of-interest setting unit 111 “a triangular region whose apex is a feature point corresponding to the right end of the mouth, a feature point corresponding to the right end of the nose, and a feature point The specified region may be set as the region of interest according to an algorithm of “setting the region of interest as the region of interest”.

The representative frame may be selected arbitrarily. For example, of the frames included in the target data range, the first frame may be selected as a representative frame.

The region of interest setting unit 111 may receive an input for specifying a region of interest and set the region of interest based on the input. In such a case, the region of interest setting unit 111 may have an input interface. The output unit 119 may cause the display device to display all or part of the moving image data, and a person may input information specifying the region of interest via the input interface while referring to the displayed information. The region of interest setting unit 111 may receive the information specifying the region of interest from the input interface, and set the region of interest based on the received information. In addition, the information which designates a region of interest is represented by the information showing a figure, for example. For example, by selecting two points in the image, the region-of-interest setting unit 111 makes a line segment connecting the two points a diagonal, and a rectangle having two sides parallel to the horizontal direction of the image The region of may be set as the region of interest.

When the pulse wave detection device 11 can recognize parts such as “forehead” and “cheeks”, the information specifying the region of interest may be information specifying the parts of the face. In such a case, the region-of-interest setting unit 111 may specify a region occupied by a part of the specified face in a representative frame, and set the region as a region of interest.

The region of interest setting unit 111 may set a plurality of regions identified by different procedures as the regions of interest.

<Brightness Data Generation Unit 112>
The luminance data generation unit 112 receives moving image data and the region of interest set by the region of interest setting unit 111 as inputs, and generates luminance data. The luminance data is time-series data of luminance values in the present embodiment.

The luminance data generation unit 112 generates variation data of luminance values obtained from the region of interest for each frame. For example, the luminance data generation unit 112 calculates the representative value of the luminance value of each color of each pixel included in the region of interest of each frame. Thereby, the luminance data generation unit 112 generates time-series data of representative values of luminance values of the respective colors. When the pixel values of the moving image data are represented by respective luminance values of the R component, the G component, and the B component, the luminance data generation unit 112 generates the luminance values of the R component, the G component, and the B component. Generate time series data of representative values. Hereinafter, the representative value of the luminance value is also referred to as a “luminance representative value”.

The representative value is a value representing the pixel value of each pixel in the region of interest. The representative value is, for example, an average value, a median value, or a mode value. The average value may be an arithmetic mean, a geometric mean or a harmonic mean.

A t-th (t = 1, 2, 3,...) Frame in moving image data is a frame t. Let R _t (i, j), G _t (i, j), and B _t (i, j) be the luminance values of the R component, G component, and B component at coordinates (i, j) of frame t, respectively. . Assuming that the region of interest is R _OI , as an example, the luminance representative value G _{t —} rep of the G component of the pixel in the region of interest is calculated by the following equation.

Here, the symbol “S” is the area of the region of interest R _OI , that is, the number of pixels included in the region of interest R _OI . This calculation formula is a calculation formula in the case where the arithmetic mean of each brightness value is adopted as the brightness representative value. In particular, when the region of interest is a rectangle and is a set of points (i, j) satisfying i _s ≦ i ≦ i _e and j _s ≦ j ≦ j _e , Gt_rep is calculated by the following equation.

The calculation method of the luminance representative value R _t _rep of the R component and the luminance representative value B _t _rep of the B component is the same.

The luminance data generation unit 112 calculates luminance representative values R _t _rep, G _t _rep, and B _t _rep for each frame included in the target data range. The luminance data generation unit 112 generates time series data R_rep, G_rep, and B_rep of luminance representative values of the respective colors by arranging the luminance representative values of the respective frames in time series.

FIG. 2 is a diagram showing an example of a graph of the luminance data G_rep. The example of FIG. 2 is an example of luminance data G_rep generated for an image of 30 seconds (ie, 901 frames including end points) acquired at a frame rate of 30 frames / second. The value on the horizontal axis is the number of the frame included in the target data range, and the value on the vertical axis is the value (Intensity) of the luminance representative value G _{t —} rep. The value of the horizontal axis of the luminance data may be represented by time (in the range of 0 seconds to 30 seconds). The luminance data is time series data indicating a pulse wave. The luminance data itself generated by the luminance data generation unit 112 is one of pulse wave information.

<Displacement Data Generation Unit 113>
The displacement data generation unit 113 receives moving image data as an input, and generates displacement data in a target data range. The displacement data is data indicating the movement of the face of the subject. The displacement data is time-series data of a change in position in the image of the face of the subject and a signal that changes in accordance with the change in orientation with respect to the image.

The displacement data generation unit 113 generates displacement data, for example, based on coordinate values of at least one feature point in an image. More specifically, for example, the displacement data generation unit 113 specifies, for each frame, the coordinate values in the lateral direction of a certain feature point. Then, the displacement data generation unit 113 generates displacement data by arranging the coordinate values in time series. The method of extracting feature points from the face may be similar to the method described above.

FIG. 3 is an example of displacement data generated according to the above example. The vertical axis is a coordinate value "x" indicating the horizontal position of the target feature point in the image. When the image has a length of 640 pixels in the horizontal direction, the range of “x” is, for example, in the range of 0 to 639.

The feature point used is, for example, one of the feature points on the contour of the nose. The nose is the part closest to the imaging device 30 when the person to be measured faces the imaging device 30. Further, the nose is far from the rotation axis when the subject rotates the face, as compared with other parts. Therefore, the coordinate values of the nose feature points are more likely to reflect changes in the face orientation than the coordinate values of other parts.

When the subject is photographed such that the lateral direction in the image matches the lateral direction for the subject, the lateral direction of the image matches the lateral direction for the subject. According to the findings of the inventors, the fluctuation of the coordinate value in the horizontal direction generates a signal (that is, noise) that interferes with the pulse wave information, rather than the fluctuation of the coordinate value in the vertical direction for the subject. It tends to be easy.

In the present disclosure, the “left and right direction” for the subject is a direction perpendicular to the front direction and the vertical direction of the face when the subject is upright. The left and right direction corresponds to, for example, a direction parallel to a straight line passing through the eyes of the subject. The lateral direction is, for example, perpendicular to the length direction of the neck of the subject.

If it is not assumed that the subject is photographed so that the lateral direction in the image matches the left-right direction for the subject, the displacement data generation unit 113 generates the right and left for the subject in the image. After identifying the direction corresponding to the direction, the coordinate value indicating the position in that direction may be acquired. The displacement data generation unit 113 detects, for example, the positions of the eyes of the subject in a typical frame, and specifies the direction of a straight line passing through both eyes as a direction corresponding to the lateral direction of the subject in the image. Just do it.

In addition, since the fluctuation of the coordinate value in the vertical direction may also be a cause of the noise, the displacement data generation unit 113 may generate the displacement data using the fluctuation of the coordinate value in the vertical direction. For example, the displacement data generation unit 113 may generate time-series data of coordinate values in the vertical direction as displacement data.

The method of generating displacement data is not limited to the above example. For example, the displacement data generation unit 113 may generate, as displacement data, time-series data indicating transition of a two-dimensional position of a certain feature point. In addition, the displacement data generation unit 113 may generate, as displacement data, time series data indicating transition of the distance from a reference position of a certain feature point. In such a case, the reference position is, for example, a point specified by the mode value of the coordinate values in the horizontal direction and the mode value of the coordinate values in the vertical direction.

The displacement data generation unit 113 may generate displacement data based on changes in the positions of the plurality of feature points. In such a case, the displacement data generation unit 113 integrates information on positions of a plurality of feature points, and generates time-series data indicating transition of one variable based on changes in positions of a plurality of feature points as displacement data. You may

For example, the displacement data generation unit 113 may extract a plurality of points from the eye, nose, and mouth feature points, and acquire the transition of the coordinate values in the horizontal direction of each of the plurality of points. Then, the displacement data generation unit 113 may generate, as displacement data, time-series data indicating the transition of the average of the coordinate values of the plurality of points.

As described in Patent Document 3, the displacement data generation unit 113 determines rotation angles of faces in each frame based on changes in positions of a plurality of feature points, and displaces time-series data indicating transition of the rotation angles. It may be generated as data.

<Pulse wave information deriving unit 114>
The pulse wave information deriving unit 114 derives pulse wave information based on the luminance data and the displacement data.

The pulse wave information deriving unit 114 in the first embodiment includes a correcting unit 1141 and a pulse rate estimating unit 1142. The correction unit 1141 corrects the luminance data based on the displacement data. Then, the pulse rate estimation unit 1142 estimates the pulse rate of the subject based on the correction data that is the luminance data after the correction.

Hereinafter, a specific example of a method of correcting the luminance data by the correction unit 1141 will be described.

The correction unit 1141 estimates, for example, the magnitude of the influence of the magnitude of the displacement on the luminance data according to the following procedure.

The correction unit 1141 obtains displacement difference data, which is data representing the difference from the reference value of the displacement data, from the displacement data. The displacement difference data is time-series data of the deviation of the data values in the displacement data from the reference value of the displacement data. When the data values in the displacement data as x (t), the reference value of the displacement data and mu _x, data value dx in the displacement difference data (t) is dx (t) = x (t) - [mu] _x. The reference value μ _x is, for example, an average value of the data values x (t) in the measurement range. The correction unit 1141 calculates dx (t) at each time to obtain time-series data of dx (t).

Further, the correction unit 1141 obtains luminance difference data, which is data representing a difference from the reference value of the luminance data, from the luminance data. The luminance difference data is time-series data of the deviation of the data values from the reference value of the luminance data. The data value in the luminance data as I (t), when the reference value of the luminance data and mu _I, the data value dI of the displacement difference data (t) is a dI (t) = I (t) - [mu] _I. The reference value μ _I is, for example, an average value of data values I (t) in the measurement range. The correction unit 1141 calculates dI (t) at each time to obtain time-series data of dI (t).

The correction unit 1141 may obtain luminance difference data for each of the luminance data for each color component. The correction unit 1141 may obtain luminance difference data for any one of the luminance data. In the following example, it is assumed that the correction unit 1141 obtains the luminance difference data dR, dG, and dB based on the luminance data of each of the R component, the G component, and the B component.

After obtaining the displacement difference data and the luminance difference data, the correction unit 1141 obtains a coefficient α that represents the relationship between the displacement difference data and the luminance difference data.

A concrete example is shown. Assuming that the relationship between the magnitude of the displacement difference and the magnitude of the effect on the luminance data (ie, the amount of change in the luminance value) is linear, data representing the change in luminance due to the displacement is It is expressed by multiplying the data value of the displacement difference data by a constant. Therefore, the correction unit 1141 obtains the value of α such that the mean square error between the value obtained by multiplying each of the data values of the displacement difference data by α and the data value of the luminance difference data is minimized. . Among the values that can be considered as a value by which each of the data values of the displacement difference data is calculated, α obtained in this way is time difference data of data values obtained by multiplying each of the data values of the displacement difference data. And the value most similar to the luminance data. α is a value representing the degree of influence of the displacement on the luminance data. α is a coefficient for converting coordinate values into luminance values.

Time-series data obtained by multiplying the data value of the displacement difference data by α is data representing a change in luminance caused by the displacement. Therefore, the correction unit 1141 obtains the correction data corrected for the change in the luminance due to the displacement by subtracting the product of the data value of the displacement difference data and α with respect to the data value of the luminance data for each frame. .

FIG. 4 is a diagram conceptually showing an example of a method of generating correction data from luminance data. In this example, the correction unit 1141 generates a difference between “brightness data” and “α times of displacement difference data” as “correction data”.

The correction data generated by the correction unit 1141 is data from the luminance data in which components resulting from changes in the position and orientation of the face are suppressed. That is, the correction data is pulse wave information from which the signal derived from the pulse wave is extracted (in other words, emphasized) of the luminance data. The pulse wave detection device 11 may output the correction data from the output unit 119.

The method of correcting the luminance data based on the displacement data is not limited to the above example, but examples other than the above example will be described later in the description of [Modification 1-2].

The pulse wave detection device 11 of the present embodiment further uses the pulse rate estimation unit 1142 to estimate the pulse rate of the subject. Hereinafter, an example of a pulse rate estimation method by the pulse rate estimation unit 1142 will be described.

The pulse rate estimation unit 1142 performs a conversion process of converting the correction data into a frequency spectrum. Examples of transformation processing to a frequency spectrum include Discrete Fourier Transform (DFT), Discrete Cosine Transform (DCT), and the like.

The pulse rate estimation unit 1142 generates a frequency spectrum by conversion processing. In the following description, a power spectrum is assumed as one aspect of a frequency spectrum.

When there is a plurality of correction data (for example, correction data of each of R component, G component, and B component), pulse rate estimating unit 1142 may generate a frequency spectrum for each of the plurality of correction data. . Alternatively, the pulse rate estimation unit 1142 integrates a plurality of correction data (for example, sums data values for each frame) to generate one time-series data, and then generates a frequency spectrum of the generated time-series data. It may be generated. Alternatively, the pulse rate estimation unit 1142 may select reliable correction data from among a plurality of correction data, and generate a frequency spectrum of the selected correction data. Examples of reliable correction data include G component correction data and correction data with the smallest variance of data values. The reason why the correction data with the smallest variance of the data values can be said to be reliable is that in the luminance data before correction, the fluctuation value of the signal derived from noise is usually larger than the fluctuation value of the signal derived from pulse wave. is there. That is, the smaller variance of the data values means that the correction can reduce the noise more appropriately.

The pulse rate estimation unit 1142 estimates the pulse based on the generated frequency spectrum. Pulse estimation is to derive an estimate of the pulse rate.

Specifically, the pulse rate estimation unit 1142 first determines a frequency region (about 50 to 180 bpm (beats per minute)) of the frequency spectrum that the subject's pulse rate can take (for example, approximately In 0.83 to 3 [Hz] [[]), the frequency at which the intensity takes the maximum value is specified. Then, the pulse rate estimation unit 1142 determines the identified frequency as the pulse rate. The unit of pulse rate may be [bpm]. The pulse rate estimation unit 1142 may convert a numerical value indicating the pulse rate in accordance with the unit. When the unit is [bpm], the value of the pulse rate may be a real number obtained by multiplying the numerical value when the unit is [Hz] by 60, or an integer closest to the real value.

When there are a plurality of frequency spectra such as frequency spectra generated for different color components, the pulse rate estimation unit 1142 may derive the pulse rate from each of the plurality of frequency spectra by the above-described method. In such a case, since a plurality of pulse rates are derived, the pulse rate estimating unit 1142 determines a representative value (average value or median value) of the plurality of derived pulse rates as an estimated pulse rate. May be Since the G component is absorbed into the blood better than the other color components, the pulse rate estimation unit 1142 gives a weighted average to the G component after giving more weight to the other color components, It may be calculated as a representative value.

The pulse wave information deriving unit 114 sends information indicating the pulse rate derived by pulse estimation to the output unit 119. The information indicating the pulse rate is, for example, an electrical signal indicating the pulse rate numerical value, text data indicating the pulse rate numerical value, and an image including display of the pulse rate numerical value.

[Modification 1-1]
A variation of the method of pulse estimation will be described. The pulse rate estimation unit 1142 derives independent components by performing independent component analysis as described in Non-Patent Document 1 on correction data relating to each color component (that is, R component, G component, B component). You may FIG. 5 is a diagram conceptually illustrating a method of performing pulse estimation using independent component analysis. As shown in FIG. 5, component data (time series) indicating three independent components can be obtained from the three correction data by independent component analysis. The pulse rate estimation unit 1142 obtains three frequency spectra ("component data (frequency)" in FIG. 5) by converting each of the obtained three component data into a frequency spectrum. The pulse rate estimation unit 1142 derives a pulse rate from the frequency spectrum obtained in this way.

The method of deriving the pulse rate from the frequency spectrum of the independent component is, for example, as follows. The pulse rate estimation unit 1142 identifies the largest peak in the frequency domain corresponding to the range that the subject's pulse rate can take in the three frequency spectra (“peak indicating pulse rate” in FIG. 5). The pulse rate estimation unit 1142 determines the frequency of the identified peak as the estimated pulse rate.

[Operation]
The operation of the pulse wave detection device 11 will be described. FIG. 6 is a flowchart showing the flow of the operation of the pulse wave detection device 11. Each process may be performed according to the order of instructions in the program when the process is performed by a device that executes the program. In the case where each process is performed by a separate device, the next process may be performed by notifying the device which has completed the process to the device that executes the next process. Note that each unit that performs processing may receive data necessary for the respective processing from the unit that generated the data or may read the data from the storage unit 110.

First, the pulse wave detection device 11 acquires moving image data from the imaging device 30 (step S11).

Then, the region of interest setting unit 111 sets a region of interest (step S12).

Then, the luminance data generation unit 112 generates luminance data from the luminance values of the pixels included in the region of interest of each frame (step S13).

The displacement data generation unit 113 generates displacement data from the position of a specific feature point of each frame (step S14).

When the luminance data and the displacement data are generated, the correction unit 1141 of the pulse wave information deriving unit 114 corrects the luminance data based on the displacement data (step S15). Then, the pulse rate estimation unit 1142 of the pulse wave information deriving unit 114 estimates the pulse rate based on the corrected data (step S16).

Then, the output unit 119 outputs information indicating the estimated pulse rate (step S17).

The pulse wave detection device 11 may generate time-series data in real time when moving image data is acquired. That is, the pulse wave detection device 11 may execute the processing for each frame in the processing from step S12 to step S14 as needed regardless of whether the moving image data has a sufficient length for analysis. In such a case, the storage unit 110 stores the luminance data and the displacement data. Then, when data for a predetermined number of frames is accumulated, the pulse wave detection device 11 may execute the processing of step S15 and subsequent steps.

The order of the above processing is an example. The order of each process can be freely changed without departing from the technical concept disclosed in the present disclosure.

[effect]
According to the pulse wave detection device 11 according to the first embodiment, the decrease in the accuracy of the pulse wave information due to the movement of the face of the person to be measured is suppressed.

The reason is that the luminance data is corrected based on the displacement data by the correction unit 1141 of the pulse wave information deriving unit 114, whereby luminance data in which data representing the pulse wave is emphasized can be obtained.

According to the tests of the inventors, a pulse rate derived using correction data generated by correction of subtracting time-series data obtained by multiplying displacement difference data by the coefficient α described above from luminance data It is known that the accuracy is better than the case where the above correction is not performed.

[Modification 1-2]
An example other than the above-described example of the method of correcting the luminance data based on the displacement data will be described.

The correction unit 1141 may obtain a value of β such that time series data obtained by raising each data value dx (t) of displacement difference data to the power of β is most similar to luminance data. The determination method of similarity is mentioned later. Then, the correction unit 1141 performs correction by subtracting the data value of time-series data obtained by raising each data value dx (t) of displacement difference data to β, from the data value of luminance data for each frame. Data may be generated.

More generally speaking, the correction unit 1141 prepares a plurality of functions f (dx (t)) for processing the data value dx (t) of the displacement difference data, and sets each of the functions to the data value dx of the displacement difference data A plurality of time series data may be generated by applying to t), and among the generated time series data, a function that generates time series data most similar to luminance data may be specified. Then, the correction unit 1141 generates correction data by subtracting time-series data generated by applying the specified function to the data value dx (t) of displacement difference data from the data value of luminance data. You may

The plurality of functions f (dx (t)) described above may be functions of a predetermined type including parameters, and the values of the parameters may be different. For example, the method already described in the description of the first embodiment corresponds to using a plurality of functions f (dx (t)) = dx (t) × α in which the value of the parameter α is different.

Another example of the function f (dx (t)) for processing the data value dx (t) is as follows.
F (dx (t)) = {dx (t)} ^ β (where “^” represents a power)
F (dx (t)) = {dx (t)} ^ β / γ
F (dx (t)) = dx (t) / (dx (t) + β)
However, the above is just an example.

Incidentally, since it is dx (t) = x (t ) -μ x, in the above description, the function for processing the data value dx of the displacement difference data (t) f (dx (t )) , the data of the displacement data It may be read as a function f (x (t)) that processes the value x (t). In that case, an example of the function f (x (t)) is as follows.
F (x (t)) = (x (t) -μ _x ) × α
F (x (t)) = (x (t) -μ _x ) ^ β
· F (x (t)) = (x (t) -μ x) / (x (t) -μ x + β)
F (x (t)) = (x (t) -μ _x ) ^ β / γ
Among the plurality of predetermined functions determined by, for example, the type of function and the value of the parameter, the correction unit 1141 generates time series data generated by processing the data value of the displacement data most similar to the luminance data. , To identify the predetermined function. The time-series data generated by processing using the specified predetermined function is time-series data indicating a change in luminance value caused by the movement of the face of the subject, and in the present disclosure, “noise time-series data” is also used. It is called. Then, the correction unit 1141 generates correction data by subtracting the noise time-series data from the data value of the luminance data.

Among the time series data (hereinafter referred to as “correction candidate time series data”) obtained by applying each of a plurality of functions to data values of displacement data, “time series data most similar to luminance data” An example of how to identify

The correction unit 1141 may specify, as “time-series data most similar to luminance data”, correction candidate time-series data with the smallest mean square error of data values in the target data range with the luminance difference data. . However, depending on the function, the correction unit 1141 may use a mean square error with the luminance data.

The correction unit 1141 selects the correction candidate time-series data in which the difference between the data value at the specific time point in the correction candidate time-series data and the data value at the specific time point in the luminance difference data is smallest. It may be specified as "time series data most similar to". The specific time point is, for example, a time point corresponding to the data value with the largest absolute value in the correction candidate time-series data. When the signal caused by noise is stronger than the signal caused by pulse wave, it is effective to refer to the data value having the largest absolute value in this way to specify the magnitude of the noise signal.

The correction unit 1141 is a correction candidate time series in which square errors with respect to a plurality of data values corresponding to a plurality of time points in the luminance displacement data of a plurality of data values corresponding to a plurality of time points in the correction candidate time series data are smallest. The data may be identified as "time-series data most similar to luminance data". The plurality of time points may be extracted, for example, according to an arbitrary algorithm for extracting the plurality of time points. For example, the correction unit 1141 may extract time points corresponding to P (where P is a set value) data values in descending order of absolute value as the plurality of time points.

[Modification 1-3]
The pulse wave detection device 11 may set a plurality of target data ranges from the acquired moving image data and calculate the pulse rate for each of the plurality of target data ranges.

For example, the pulse wave detection device 11 sets a range from the first frame to the Nth (N is a set value) frame of the acquired moving image data as a first target data range, k × M. A range from the first frame (M is a set value, k = 1, 2,...) To the N + k × Mth frame may be set as the k-th target data range. Then, pulse wave information may be derived for each target data range, and a total of k estimated pulse rates may be calculated.

Then, the pulse wave detection device 11 may output the representative value of the estimated values of the k pulse rates as the estimated pulse rate. There is no limitation on the type of representative value, but it is an average value or median value as an example. The pulse wave detection device 11 may have a mode value as the representative value if the estimated values of the k pulse rates are integer values.

In such a configuration, the pulse wave detection device 11 may include the outlier removal unit 115 as illustrated in FIG. 7. FIG. 7 shows the configuration of the pulse wave detection device 11 a including the outlier removal unit 115. The configuration of the pulse wave detection device 11 a may be the same as the configuration of the pulse wave detection device 11 except that the outlier removal unit 115 is provided.

Outlier removal section 115 removes an estimated value specified as an outlier from a plurality of calculated estimated values. The method of identifying outliers may be a method already known. The pulse wave detection device 11a obtains a representative value of the remaining estimated values obtained by removing outliers from the plurality of calculated estimated values, and outputs the representative value as an estimated pulse rate. By removing outliers, the pulse rate can be estimated more accurately.

<< Second Embodiment >>
Hereinafter, as a second embodiment, an embodiment in which the pulse wave detection device derives pulse wave information using a plurality of regions of interest will be described.

FIG. 8 is a block diagram showing the configuration of the pulse wave detection device 12 according to the second embodiment. Similar to the pulse wave detection device 11a according to the modification of the first embodiment, the pulse wave detection device 12 includes a storage unit 110, a region of interest setting unit 111, a luminance data generation unit 112, a displacement data generation unit 113, and an outlier removal unit. 115, and an output unit 119. The pulse wave detection device 12 includes a pulse wave information deriving unit 124 instead of the pulse wave information deriving unit 114.

The pulse wave information deriving unit 124 includes a correcting unit 1241, an associating unit 1242, a combining unit 1243, and a pulse rate estimating unit 1244.

The function of the correction unit 1241 may be the same as the function of the correction unit 1141 of the first embodiment.

However, in the second embodiment, the region of interest setting unit 111 sets a plurality of regions of interest. The pulse wave detection device 12 then generates luminance data and displacement data for each of the plurality of regions of interest. The correction unit 1241 of the pulse wave information derivation unit 124 generates correction data for each of the plurality of regions of interest.

The correction data of each of the plurality of regions of interest are integrated into one time series data by the processing of the associating unit 1242 and the combining unit 1243 described below.

<Association unit 1242>
The associating unit 1242 associates, with each of the plurality of regions of interest, the degree of reliability indicating the degree of reliability. The reliability is expressed, for example, as a numerical value, and the higher the value of reliability, the higher the reliability.

The reliability indicated by the reliability is the reliability as an area in which pulse wave information can be accurately obtained. That is, the reliability indicates how reliable the variation of the luminance value obtained from the region of interest to which the reliability is associated can be as data representing a pulse wave.

Specifically, the associating unit 1242 associates the degrees of reliability with the respective regions of interest such that the regions of interest in which the variance of the data values of the correction data is smaller become higher in reliability. The size of the distribution of data values, for example, the evaluation formula Σ _{t = 1 → t_max} distribution value | only to be evaluated by the ^{_{2 / t_max | V E -V t}} . However, t_max is the number of frames included in the target data _{range, V t (t = 1,2,3,} ..., t_max) is t-th frame of data values, _{V E} is the reference value of the data values (e.g. _{V t} Mean, median or mode).

Further, the associating unit 1242 may uniformly associate the lowest value (for example, “0”) as the reliability with respect to the region of interest uniformly with the variance exceeding a predetermined reference (for example, the variance value exceeding the predetermined value). .

The associating unit 1242 may use a method of associating the first numerical value D1 when the variance value exceeds the predetermined threshold and associating the second numerical value D2 when the variance value does not exceed the predetermined threshold. In such a case, D1 is smaller than D2.

The association unit 1242 may normalize the respective reliabilities so that the sum of the reliabilities of the respective regions of interest is 1 after associating the reliabilities with all the regions of interest.

<Composition unit 1243>
The combining unit 1243 combines one piece of time series data from correction data of a plurality of regions of interest based on the reliability of the plurality of regions of interest.

If there is correction data for each color component, the combining unit 1243 may combine the correction data for each color component. That is, for example, the combining unit 1243 may generate one each of R component time-series data, G component time-series data, and B component time-series data.

An example of a specific processing flow of composition using correction data of a plurality of regions of interest regarding one color component will be described below.

First, the combining unit 1243 weights the correction data of each region of interest based on the reliability. Specifically, for example, the combining unit 1243 obtains weighted correction data by multiplying the data value of the correction data of each region of interest by the reliability of the region of interest.

Then, the combining unit 1243 combines the weighted correction data. For example, the combining unit 1243 obtains one time-series data by adding data values of weighted correction data for each frame. Alternatively, the combining unit 1243 may obtain one time-series data by calculating an average (regardless of the type of the average) of each of the weighted correction data.

<Pulse rate estimation unit 1244>
The function of the pulse rate estimation unit 1244 may be the same as that of the pulse rate estimation unit 1142 of the first embodiment.

[effect]
According to the pulse wave detection device 12 according to the second embodiment, in addition to the effects of the first embodiment, it is possible to acquire pulse wave information with higher accuracy. The reason is that the correlation unit 1242 relates the reliability to the region of interest so that the contribution of data obtained from the region of interest having high reliability is large, and the synthesis unit 1243 determines the time series data of pulse waves based on the reliability. Is generated. The time-series data of this pulse wave is one of pulse wave information. In addition, it is expected that the estimated value of the pulse rate calculated based on the time series data of this pulse wave is more accurate.

<< Third Embodiment >>
As a third embodiment of the present invention, an embodiment will be described in which the method of deriving the pulse rate is different from that of the first embodiment.

FIG. 9 is a block diagram showing the configuration of the pulse wave detection device 13 according to the third embodiment. Similar to the pulse wave detection device 11 according to the first embodiment, the pulse wave detection device 13 includes a storage unit 110, a region of interest setting unit 111, a luminance data generation unit 112, a displacement data generation unit 113, and an output unit 119. The pulse wave detection device 13 includes a pulse wave information deriving unit 134 instead of the pulse wave information deriving unit 114.

The pulse wave information deriving unit 134 derives pulse wave information from the luminance data and the displacement data. The pulse wave information deriving unit 134 includes a data analyzing unit 1341 and a pulse rate estimating unit 1342.

The data analysis unit 1341 performs independent component analysis on a plurality of luminance value data obtained in the same time range to generate a plurality of component data.

The pulse rate estimation unit 1342 estimates the pulse rate based on the component data and displacement data generated by the data analysis unit 1341.

In the method of estimating the pulse rate using component data obtained by independent component analysis, as disclosed in Non-Patent Document 1, it is usually within the range of values that can be taken as pulse rate in any component data. The peak frequency is determined as the pulse rate. If more than one peak is found in multiple component data in total within the range of values that can be taken as a pulse rate, it is necessary to select one of the frequencies of any peak as an estimate of the pulse rate. is there. However, in such a case, it may not be self-evident which peak corresponds to the pulse rate. One of the plurality of peaks is a peak derived from the noise component. If such a peak is determined as the pulse rate, an incorrect estimate will be output.

The pulse rate estimation unit 1342 according to the present embodiment identifies a peak estimated to be a noise-derived peak based on the frequency spectrum of displacement data, and estimates a pulse rate from a peak other than the peak. Note that the data analysis unit 1341 may generate a frequency spectrum of displacement data.

A specific example of the method of estimating the pulse rate will be described using the example of FIG.

FIG. 10 shows an example of component data of three independent components. The pulse rate estimating unit 1342 extracts a peak having a predetermined intensity or more in the frequency range corresponding to the range that can be taken by the pulse rate of the subject, for example, for the three component data. As an example, it is assumed that one peak is extracted from each of the component data of the second component and the component data of the third component. The two peaks (denoted respectively as “first candidate” and “second candidate”) are candidates for the peak indicating the pulse rate.

The pulse rate estimation unit 1342 compares the peak in the frequency spectrum of the displacement data with the plurality of extracted peaks. Then, the pulse rate estimation unit 1342 estimates that the candidate having the same frequency as the frequency of the peak present in the frequency spectrum of the displacement data is a peak derived from noise, and excludes it from the candidate of the peak indicating the pulse rate. For example, as shown in FIG. 10, it is assumed that there is a peak of the same frequency as the second candidate frequency in the frequency spectrum of displacement data. In this case, the pulse rate estimation unit 1342 excludes the second candidate from the candidate of the peak indicating the pulse rate. Therefore, the pulse rate estimation unit 1342 selects the first candidate as a peak indicating the pulse rate.

Two peaks of the same frequency do not necessarily mean only two peaks whose frequencies completely match. The pulse rate estimation unit 1342 may regard two peaks whose frequency difference is within a predetermined threshold as “peaks of the same frequency”. The pulse rate estimation unit 1342 may regard the peak of the frequency closest to the frequency of the peak in the frequency spectrum of the displacement data among the plurality of candidates as the “peak of the same frequency”.

If there are three or more peak candidates indicating the pulse rate, etc. and two or more candidates still remain even if one or more candidates are excluded from the peak candidates indicating the pulse rate, then the pulse rate estimation is performed. The unit 1342 may determine one peak from the remaining candidates as a peak indicating a pulse rate. For example, the pulse rate estimating unit 1342 may determine the peak with the largest intensity among the remaining candidates as the peak indicating the pulse rate.

The pulse rate estimation unit 1342 determines the frequency of the peak determined as the peak indicating the pulse rate as the estimated pulse rate.

The pulse wave information deriving unit 134 sends information indicating the determined pulse rate to the output unit 119. The format of the information indicating the pulse rate may be similar to the example shown in the first embodiment.

[effect]
According to the pulse wave detection device 13 according to the third embodiment, as in the first embodiment, the decrease in the accuracy of the pulse wave information due to the movement of the face of the person to be measured is suppressed. The reason is that the pulse rate estimating unit 1342 of the pulse wave information deriving unit 114 uses the displacement data to indicate a peak of the pulse rate that is different from the frequency of the peak derived from the noise component caused by the movement of the face. It is because it selects as. That is, it is possible to prevent the peak derived from the noise component from being determined as the peak indicating the pulse rate. As a result, the accuracy of the output pulse wave information is improved.

The process by the pulse rate estimation unit 1342 is a process of excluding a component derived from noise among a plurality of components, in other words, a process of selecting or extracting a signal component derived from a pulse wave from the plurality of components. is there.

The pulse wave detection device 13 according to the third embodiment also calculates the pulse rate for a plurality of target data ranges as in the modification 1-3 of the pulse wave detection device according to the first embodiment. May be The pulse wave detection device 13 according to the third embodiment may also have the outlier removal unit 115 like the pulse wave detection device 11a.

<< Fourth Embodiment >>
FIG. 11 is a block diagram showing the configuration of a pulse wave detection device 10 according to an embodiment of the present invention. The pulse wave detection apparatus 10 includes a region of interest setting unit 101, a luminance data generation unit 102, a displacement data generation unit 103, a pulse wave component extraction unit 104, and an output unit 105.

The region of interest setting unit 101 sets the region of interest of the face of the subject in the time-series image in which the face of the subject is captured. The region of interest setting unit 111 in each of the above embodiments is an example of the region of interest setting unit 101.

The luminance data generation unit 102 generates luminance data which is time-series data of representative values of luminance values of predetermined color components of pixels in the region of interest. The luminance data generation unit 112 in each of the above embodiments is an example of the luminance data generation unit 102.

The displacement data generation unit 103 generates displacement data indicating movement of the face of the subject based on the time-series image. The displacement data generation unit 113 in each of the above embodiments is an example of the displacement data generation unit 103.

The pulse wave component extraction unit 104 extracts the signal component caused by the pulse wave in the luminance data using the displacement data. Correcting the luminance data based on the displacement data corresponds to extracting a signal component caused by the pulse wave. Selecting an estimated value of pulse rate by referring to displacement data is also equivalent to extracting a signal component caused by a pulse wave.

The pulse wave information deriving units 114 124, and 134 in the above-described embodiments are an example of the pulse wave component extracting unit 104.

The output unit 105 outputs pulse wave information based on the extracted signal component. The pulse wave information may be the extracted signal component itself or an estimated value of the pulse rate calculated based on the extracted signal component. The output unit 119 in each of the above-described embodiments is an example of the output unit 105.

FIG. 12 is a flowchart showing the flow of the operation of the pulse wave detection device 10.

First, the region of interest setting unit 101 sets the region of interest of the face of the subject in the time-series image (step S101).

Next, the luminance data generation unit 102 generates luminance data which is time-series data of representative values of luminance values of predetermined color components of the pixels in the region of interest (step S102).

Also, the displacement data generation unit 103 generates displacement data indicating the movement of the face of the person to be measured based on the time-series image (step S103).

Then, the pulse wave component extraction unit 104 extracts the signal component caused by the pulse wave in the luminance data using the displacement data (step S104).

Then, the output unit 105 outputs pulse wave information based on the extracted signal component (step S105).

The processes of step S102 and step S103 may be performed in reverse order or may be performed in parallel.

According to the pulse wave detection device 10 of the present embodiment, it is possible to more accurately acquire pulse wave information from a captured image of a subject. The reason is that the pulse wave component extraction unit 104 extracts the information resulting from the pulse wave from the luminance data based on the data indicating the movement of the face.

<Configuration of Hardware for Implementing Each Part of Embodiment>
As described above, in each of the embodiments of the present invention described above, each component of each device indicates a block of function units.

The processing of each component may be realized by, for example, a computer system reading and executing a program stored in a computer-readable storage medium that causes the computer system to execute the processing. The “computer-readable storage medium” is, for example, a portable medium such as an optical disc, a magnetic disc, a magneto-optical disc, and a nonvolatile semiconductor memory, and a ROM (Read Only Memory) and a hard disc incorporated in a computer system. It is a storage device. The “computer readable storage medium” is one that holds a program dynamically for a short time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, such as volatile memory in a computer system that corresponds to a server or a client in that case, the one that holds a program temporarily is also included. Further, the program may be for realizing a part of the functions described above, and may be capable of realizing the functions described above in combination with a program already stored in a computer system.

The “computer system” is a system including a computer 900 as shown in FIG. 13 as an example. The computer 900 includes the following configuration.
・ One or more CPUs (Central Processing Unit) 901
ROM 902
RAM (Random Access Memory) 903
· Program 904A loaded into RAM 903 and stored information 904B
A storage device 905 for storing the program 904A and the stored information 904B
. Drive device 907 for reading and writing the storage medium 906
Communication interface 908 connected to communication network 909
・ Input / output interface 910 for data input / output
.Bus 911 connecting each component

For example, each component of each device in each embodiment is realized by the CPU 901 loading and executing a program 904A that implements the function of the component to the RAM 903. A program 904A for realizing the function of each component of each device is stored in advance in, for example, the storage device 905 or the ROM 902. Then, the CPU 901 reads the program 904A as necessary. The storage device 905 is, for example, a hard disk. The program 904A may be supplied to the CPU 901 via the communication network 909, may be stored in advance in the storage medium 906, may be read by the drive device 907, and may be supplied to the CPU 901. The storage medium 906 is, for example, a portable medium such as an optical disc, a magnetic disc, a magneto-optical disc, and a nonvolatile semiconductor memory.

There are various modifications in the implementation method of each device. For example, each device may be realized by possible combination of separate computer 900 and program for each component. In addition, a plurality of components included in each device may be realized by a possible combination of one computer 900 and a program.

In addition, part or all of each component of each device may be realized by another general purpose or dedicated circuit, a computer or the like, or a combination thereof. These may be configured by a single chip or may be configured by a plurality of chips connected via a bus.

When a part or all of each component of each device is realized by a plurality of computers, circuits, etc., the plurality of computers, circuits, etc. may be centralized or distributed. For example, a computer, a circuit, etc. may be realized as a form in which each is connected via a communication network, such as a client and server system, a cloud computing system, and the like.

Some or all of the above embodiments may be described as in the following appendices, but is not limited thereto.

<< Appendix >>
[Supplementary Note 1]
A region-of-interest setting unit configured to set a region of interest of the face of the subject for a time-series image of the face of the subject being photographed;
Luminance data generation means for generating luminance data which is time-series data of representative values of luminance values of predetermined color components of pixels in the region of interest;
Displacement data generation means for generating displacement data indicating movement of the face of the subject based on the time-series image;
Pulse wave component extraction means for extracting a signal component attributable to a pulse wave in the luminance data using the displacement data;
An output unit that outputs pulse wave information based on the extracted signal component;
Pulse wave detection device comprising:
[Supplementary Note 2]
The displacement data generation unit according to claim 1, wherein the displacement data generation unit generates the displacement data based on a change in one reference direction set for the time-series image of the position of the feature point detected from the face. Pulse wave detection device.
[Supplementary Note 3]
The pulse wave detection device according to claim 2, wherein the displacement data generation unit sets the reference direction based on the direction of the face.
[Supplementary Note 4]
The pulse wave detection device according to Supplementary Note 2 or 3, wherein the displacement data generation means sets a direction corresponding to the lateral direction for the subject as the reference direction.
[Supplementary Note 5]
The pulse wave detection device according to any one of appendices 1 to 4, wherein the displacement data generation unit generates the displacement data using a position of a feature point in the nose of the subject.
[Supplementary Note 6]
The pulse wave component extraction unit generates noise time-series data indicating a change in luminance value caused by the movement of the face of the subject using the displacement data and the luminance data, and the noise time-series data Correcting the luminance data by subtracting the data value of the luminance data from the data value of the luminance data,
The pulse wave detection device according to any one of appendices 1 to 5.
[Supplementary Note 7]
The pulse wave component extraction means identifies the predetermined function whose time series data generated by processing is most similar to the luminance data among a plurality of predetermined functions for processing data values of the displacement data, and Generating time series data generated by processing using the specified predetermined function as the noise time series data;
The pulse wave detection device according to appendix 6.
[Supplementary Note 8]
The predetermined function is a function of subtracting a reference value of displacement from a data value of the displacement data and then multiplying the result by a coefficient.
The pulse wave component extraction means calculates a mean square error between time series data generated by processing using the predetermined function and luminance difference data indicating a difference from the luminance reference value of the luminance data. By calculating the value of the coefficient which becomes the smallest, the predetermined function whose time series data generated by processing is most similar to the luminance data is specified.
The pulse wave detection device according to appendix 6.
[Supplementary Note 9]
The luminance data generation unit generates the luminance data from each of the plurality of regions of interest;
The pulse wave detection device
Association means for associating, with each of the plurality of luminance data corrected by the pulse wave component extraction means, a reliability based on the magnitude of dispersion of data values;
Each of the plurality of corrected luminance data is weighted based on the reliability, and data values of the plurality of weighted luminance data are added at each time to generate the combined luminance data. Synthesis means,
The pulse wave detection device according to any one of appendices 6 to 8, comprising:
[Supplementary Note 10]
The luminance data generation unit generates the luminance data for each of a plurality of color components,
The pulse wave component extraction means applies independent component analysis to the luminance data of a plurality of color components to generate a plurality of time series data respectively indicating different signal components, and a plurality of the plurality of time series data are generated. Identifying the frequency of the pulse rate as a pulse rate candidate, excluding the same frequency as the frequency of the peak present in the frequency spectrum of the displacement data from the pulse rate candidates, and estimating the pulse rate from the candidates not excluded.
The pulse wave detection device according to any one of appendices 1 to 5.
[Supplementary Note 11]
The region of interest of the face of the subject is set in the time-series image in which the face of the subject is photographed,
Generating luminance data which is time-series data of representative values of luminance values of predetermined color components of pixels in the region of interest;
Generating displacement data indicating movement of the face of the subject based on the time-series image;
The displacement data is used to extract a signal component attributable to a pulse wave in the luminance data,
Outputting pulse wave information based on the extracted signal component,
Pulse wave detection method.
[Supplementary Note 12]
15. The pulse wave detection method according to appendix 11, wherein the displacement data is generated based on a change in one reference direction set for the time-series image of the position of the feature point detected from the face.
[Supplementary Note 13]
15. The pulse wave detection method according to appendix 12, wherein the reference direction is set based on the direction of the face.
[Supplementary Note 14]
The pulse wave detection method according to appendix 12 or 13, wherein a direction corresponding to the lateral direction for the subject is set as the reference direction.
[Supplementary Note 15]
The pulse wave detection method according to any one of appendices 11 to 14, wherein the displacement data is generated using a position of a feature point in the nose of the subject.
[Supplementary Note 16]
Using the displacement data and the luminance data, noise time-series data indicating a change in luminance value caused by the movement of the face of the subject is generated, and the data value of the noise time-series data is calculated as the luminance data Correct the luminance data by subtracting from the data values of
The pulse wave detection method according to any one of appendices 11 to 15.
[Supplementary Note 17]
Among a plurality of predetermined functions for processing data values of the displacement data, the predetermined function whose time series data generated by processing is most similar to the luminance data is specified, and the specified predetermined function is used. Generating time series data generated by the processing as the noise time series data,
The pulse wave detection method according to appendix 16.
[Supplementary Note 18]
The predetermined function is a function of subtracting a reference value of displacement from a data value of the displacement data and then multiplying the result by a coefficient.
The coefficients such that the mean square error between the time series data generated by processing using the predetermined function and the luminance difference data indicating the difference from the luminance reference value of the luminance data is minimized. The predetermined function whose time series data generated by processing is most similar to the luminance data is calculated by calculating the value of
The pulse wave detection method according to appendix 17.
[Supplementary Note 19]
Generating the luminance data from each of a plurality of the regions of interest;
Relating each of a plurality of the luminance data corrected using the displacement data and the luminance data to a reliability based on the magnitude of the dispersion of the data value;
Each of the plurality of corrected luminance data is weighted based on the reliability, and data values of the plurality of weighted luminance data are added at each time to generate the combined luminance data. ,
The pulse wave detection method according to any one of appendices 16 to 18.
[Supplementary Note 20]
Generating the luminance data for each of a plurality of color components;
By applying independent component analysis to the luminance data of a plurality of color components, a plurality of time series data respectively indicating different signal components are generated, and a plurality of frequencies are set as pulse rate candidates from the plurality of time series data. Identifying and excluding from the pulse rate candidate the same frequency as the frequency of the peak present in the frequency spectrum of the displacement data, and estimating the pulse rate from the not excluded candidates;
The pulse wave detection method according to any one of appendices 11 to 15.
[Supplementary Note 21]
On the computer
A region of interest setting process of setting a region of interest of the face of the subject with respect to a time-series image of the face of the subject being photographed;
Luminance data generation processing for generating luminance data which is time-series data of representative values of luminance values of predetermined color components of pixels in the region of interest;
Displacement data generation processing for generating displacement data indicating movement of the face of the subject based on the time-series image;
Pulse wave component extraction processing for extracting a signal component attributable to a pulse wave in the luminance data using the displacement data;
An output process for outputting pulse wave information based on the extracted signal component;
A non-transitory computer readable storage medium storing a program for executing the program.
[Supplementary Note 22]
The displacement data generation process according to claim 21, wherein the displacement data generation process generates the displacement data based on a change in one reference direction set with respect to the time-series image of the position of the feature point detected from the face. Storage medium.
[Supplementary Note 23]
The storage medium according to Appendix 22, wherein the displacement data generation process sets the reference direction based on the direction of the face.
[Supplementary Note 24]
The storage medium according to Appendix 22 or 23, wherein the displacement data generation process sets a direction corresponding to the lateral direction for the subject as the reference direction.
[Supplementary Note 25]
24. The storage medium according to any one of appendices 21 to 24, wherein the displacement data generation process generates the displacement data using a position of a feature point in the nose of the subject.
[Supplementary Note 26]
The pulse wave component extraction process generates noise time-series data indicating a change in luminance value caused by the movement of the face of the subject using the displacement data and the luminance data, and the noise time-series data Correcting the luminance data by subtracting the data value of the luminance data from the data value of the luminance data,
24. A storage medium according to any one of appendices 21-25.
[Supplementary Note 27]
The pulse wave component extraction process specifies, among a plurality of predetermined functions for processing data values of the displacement data, the predetermined function in which time series data generated by processing is most similar to the luminance data, Generating time series data generated by processing using the specified predetermined function as the noise time series data;
24. A storage medium according to appendix 26.
[Supplementary Note 28]
The predetermined function is a function of subtracting a reference value of displacement from a data value of the displacement data and then multiplying the result by a coefficient.
In the pulse wave component extraction process, a mean square error between time-series data generated by processing using the predetermined function and luminance difference data indicating a difference from the luminance reference value of the luminance data is By calculating the value of the coefficient which becomes the smallest, the predetermined function whose time series data generated by processing is most similar to the luminance data is specified.
24. The storage medium according to appendix 27.
[Supplementary Note 29]
The luminance data generation process generates the luminance data from each of a plurality of the regions of interest,
The program is stored in the computer
An associating process for associating, with each of the plurality of luminance data corrected by the pulse wave component extracting process, a reliability based on a magnitude of dispersion of data values;
Each of the plurality of corrected luminance data is weighted based on the reliability, and data values of the plurality of weighted luminance data are added at each time to generate the combined luminance data. Composition processing,
24. The storage medium according to any one of appendices 26-28, wherein
[Supplementary Note 30]
The luminance data generation process generates the luminance data for each of a plurality of color components,
The pulse wave component extraction processing applies independent component analysis to the luminance data of a plurality of color components to generate a plurality of time series data respectively indicating different signal components, and a plurality of the plurality of time series data are generated. Identifying the frequency of the pulse rate as a pulse rate candidate, excluding the same frequency as the frequency of the peak present in the frequency spectrum of the displacement data from the pulse rate candidates, and estimating the pulse rate from the candidates not excluded.
24. A storage medium according to any one of appendices 21-25.

The present invention is not limited to the embodiments described above. The configurations and details of the embodiment described above can be modified in various ways that can be understood by those skilled in the art within the scope of the present invention.

Reference numerals 10 to 13 and 11a pulse wave detection device 30 imaging device 101 region of interest setting unit 102 luminance data generation unit 103 displacement data generation unit 104 pulse wave component extraction unit 105 output unit 110 storage unit 111 interest region setting unit 112 luminance data generation unit 113 Displacement data generation unit 114 Pulse wave information derivation unit 1141 Correction unit 1142 Pulse rate estimation unit 115 Outlier removal unit 119 Output unit 124 Pulse wave information derivation unit 1241 Correction unit 1242 Association unit 1243 Synthesis unit 1244 Pulse rate estimation unit 134 Pulse wave information Derivation unit 1341 Data analysis unit 1342 Pulse rate estimation unit 900 Computer 901 CPU
902 ROM
903 RAM
904A program 904B storage information 905 storage device 906 storage medium 907 drive device 908 communication interface 909 communication network 910 input / output interface 911 bus

Claims (30)

1. A region-of-interest setting unit configured to set a region of interest of the face of the subject for a time-series image of the face of the subject being photographed;
   Luminance data generation means for generating luminance data which is time-series data of representative values of luminance values of predetermined color components of pixels in the region of interest;
   Displacement data generation means for generating displacement data indicating movement of the face of the subject based on the time-series image;
   Pulse wave component extraction means for extracting a signal component attributable to a pulse wave in the luminance data using the displacement data;
   An output unit that outputs pulse wave information based on the extracted signal component;
   Pulse wave detection device comprising:
2. The displacement data generation means generates the displacement data based on a change in one reference direction set for the time-series image of the position of the feature point detected from the face. Pulse wave detection device as described.
3. The pulse wave detection device according to claim 2, wherein the displacement data generation unit sets the reference direction based on the direction of the face.
4. The pulse wave detection device according to claim 2, wherein the displacement data generation unit sets a direction corresponding to a lateral direction for the subject as the reference direction.
5. The pulse wave detection device according to any one of claims 1 to 4, wherein the displacement data generation unit generates the displacement data using a position of a feature point in the nose of the subject.
6. The pulse wave component extraction unit generates noise time-series data indicating a change in luminance value caused by the movement of the face of the subject using the displacement data and the luminance data, and the noise time-series data Correcting the luminance data by subtracting the data value of the luminance data from the data value of the luminance data,
   The pulse wave detection device according to any one of claims 1 to 5.
7. The pulse wave component extraction means identifies the predetermined function whose time series data generated by processing is most similar to the luminance data among a plurality of predetermined functions for processing data values of the displacement data, and Generating time series data generated by processing using the specified predetermined function as the noise time series data;
   The pulse wave detection device according to claim 6.
8. The predetermined function is a function of subtracting a reference value of displacement from a data value of the displacement data and then multiplying the result by a coefficient.
   The pulse wave component extraction means calculates a mean square error between time series data generated by processing using the predetermined function and luminance difference data indicating a difference from the luminance reference value of the luminance data. By calculating the value of the coefficient which becomes the smallest, the predetermined function whose time series data generated by processing is most similar to the luminance data is specified.
   The pulse wave detection device according to claim 6.
9. The luminance data generation unit generates the luminance data from each of the plurality of regions of interest;
   The pulse wave detection device
   Association means for associating, with each of the plurality of luminance data corrected by the pulse wave component extraction means, a reliability based on the magnitude of dispersion of data values;
   Each of the plurality of corrected luminance data is weighted based on the reliability, and data values of the plurality of weighted luminance data are added at each time to generate the combined luminance data. Synthesis means,
   The pulse wave detection device according to any one of claims 6 to 8, comprising:
10. The luminance data generation unit generates the luminance data for each of a plurality of color components,
    The pulse wave component extraction means applies independent component analysis to the luminance data of a plurality of color components to generate a plurality of time series data respectively indicating different signal components, and a plurality of the plurality of time series data are generated. Identifying the frequency of the pulse rate as a pulse rate candidate, excluding the same frequency as the frequency of the peak present in the frequency spectrum of the displacement data from the pulse rate candidates, and estimating the pulse rate from the candidates not excluded.
    The pulse wave detection device according to any one of claims 1 to 5.
11. The region of interest of the face of the subject is set in the time-series image in which the face of the subject is photographed,
    Generating luminance data which is time-series data of representative values of luminance values of predetermined color components of pixels in the region of interest;
    Generating displacement data indicating movement of the face of the subject based on the time-series image;
    The displacement data is used to extract a signal component attributable to a pulse wave in the luminance data,
    Outputting pulse wave information based on the extracted signal component,
    Pulse wave detection method.
12. The pulse wave detection method according to claim 11, wherein the displacement data is generated based on a change in one reference direction set with respect to the time-series image of the position of the feature point detected from the face.
13. The pulse wave detection method according to claim 12, wherein the reference direction is set based on the direction of the face.
14. The pulse wave detection method according to claim 12, wherein a direction corresponding to the lateral direction for the subject is set as the reference direction.
15. The pulse wave detection method according to any one of claims 11 to 14, wherein the displacement data is generated using the position of the feature point in the nose of the subject.
16. Using the displacement data and the luminance data, noise time-series data indicating a change in luminance value caused by the movement of the face of the subject is generated, and the data value of the noise time-series data is calculated as the luminance data Correct the luminance data by subtracting from the data values of
    The pulse wave detection method according to any one of claims 11 to 15.
17. Among a plurality of predetermined functions for processing data values of the displacement data, the predetermined function whose time series data generated by processing is most similar to the luminance data is specified, and the specified predetermined function is used. Generating time series data generated by the processing as the noise time series data,
    The pulse wave detection method according to claim 16.
18. The predetermined function is a function of subtracting a reference value of displacement from a data value of the displacement data and then multiplying the result by a coefficient.
    The coefficients such that the mean square error between the time series data generated by processing using the predetermined function and the luminance difference data indicating the difference from the luminance reference value of the luminance data is minimized. The predetermined function whose time series data generated by processing is most similar to the luminance data is calculated by calculating the value of
    The pulse wave detection method according to claim 17.
19. Generating the luminance data from each of a plurality of the regions of interest;
    Relating each of a plurality of the luminance data corrected using the displacement data and the luminance data to a reliability based on the magnitude of the dispersion of the data value;
    Each of the plurality of corrected luminance data is weighted based on the reliability, and data values of the plurality of weighted luminance data are added at each time to generate the combined luminance data. ,
    The pulse wave detection method according to any one of claims 16 to 18.
20. Generating the luminance data for each of a plurality of color components;
    By applying independent component analysis to the luminance data of a plurality of color components, a plurality of time series data respectively indicating different signal components are generated, and a plurality of frequencies are set as pulse rate candidates from the plurality of time series data. Identifying and excluding from the pulse rate candidate the same frequency as the frequency of the peak present in the frequency spectrum of the displacement data, and estimating the pulse rate from the not excluded candidates;
    The pulse wave detection method according to any one of claims 11 to 15.
21. On the computer
    A region of interest setting process of setting a region of interest of the face of the subject with respect to a time-series image of the face of the subject being photographed;
    Luminance data generation processing for generating luminance data which is time-series data of representative values of luminance values of predetermined color components of pixels in the region of interest;
    Displacement data generation processing for generating displacement data indicating movement of the face of the subject based on the time-series image;
    Pulse wave component extraction processing for extracting a signal component attributable to a pulse wave in the luminance data using the displacement data;
    An output process for outputting pulse wave information based on the extracted signal component;
    A non-transitory computer readable storage medium storing a program for executing the program.
22. 22. The displacement data generation process according to claim 21, wherein the displacement data generation processing generates the displacement data based on a change in one reference direction set for the time-series image of the position of the feature point detected from the face. Storage medium as described.
23. The storage medium according to claim 22, wherein the displacement data generation process sets the reference direction based on an orientation of the face.
24. The storage medium according to claim 22 or 23, wherein the displacement data generation process sets a direction corresponding to the lateral direction for the subject as the reference direction.
25. The storage medium according to any one of claims 21 to 24, wherein the displacement data generation processing generates the displacement data using a position of a feature point in the nose of the subject.
26. The pulse wave component extraction process generates noise time-series data indicating a change in luminance value caused by the movement of the face of the subject using the displacement data and the luminance data, and the noise time-series data Correcting the luminance data by subtracting the data value of the luminance data from the data value of the luminance data,
    The storage medium according to any one of claims 21 to 25.
27. The pulse wave component extraction process specifies, among a plurality of predetermined functions for processing data values of the displacement data, the predetermined function in which time series data generated by processing is most similar to the luminance data, Generating time series data generated by processing using the specified predetermined function as the noise time series data;
    The storage medium according to claim 26.
28. The predetermined function is a function of subtracting a reference value of displacement from a data value of the displacement data and then multiplying the result by a coefficient.
    In the pulse wave component extraction process, a mean square error between time-series data generated by processing using the predetermined function and luminance difference data indicating a difference from the luminance reference value of the luminance data is By calculating the value of the coefficient which becomes the smallest, the predetermined function whose time series data generated by processing is most similar to the luminance data is specified.
    The storage medium according to claim 27.
29. The luminance data generation process generates the luminance data from each of a plurality of the regions of interest,
    The program is stored in the computer
    An associating process for associating, with each of the plurality of luminance data corrected by the pulse wave component extracting process, a reliability based on a magnitude of dispersion of data values;
    Each of the plurality of corrected luminance data is weighted based on the reliability, and data values of the plurality of weighted luminance data are added at each time to generate the combined luminance data. Composition processing,
    The storage medium according to any one of claims 26 to 28, wherein the storage medium is a storage medium.
30. The luminance data generation process generates the luminance data for each of a plurality of color components,
    The pulse wave component extraction processing applies independent component analysis to the luminance data of a plurality of color components to generate a plurality of time series data respectively indicating different signal components, and a plurality of the plurality of time series data are generated. Identifying the frequency of the pulse rate as a pulse rate candidate, excluding the same frequency as the frequency of the peak present in the frequency spectrum of the displacement data from the pulse rate candidates, and estimating the pulse rate from the candidates not excluded.
    The storage medium according to any one of claims 21 to 25.

Images