JP6521845B2 - Device and method for measuring periodic fluctuation linked to heart beat - Google Patents

Description

  The present invention relates to a measuring apparatus and method for measuring periodic fluctuations linked to the heart rate, such as the subject's pulse rate and pulse wave signals, from the image of the subject's face, and the light striking the face changes even if the face moves. However, it is possible to measure accurately.

  Techniques for measuring a subject's pulse rate and respiration rate from an image of the subject's face are known in the prior art. For example, in Patent Document 1 below, the average luminance of the region of interest (ROI) of the face is calculated from the face image of the subject imaged by the imaging means, and this luminance is used as the position or face direction of the region of interest. It is described that the subject's pulse rate is calculated from the waveform of the temporal change of the luminance corrected based on the correction.

  Also, in Non-Patent Document 1 below, a video signal obtained by capturing the face ROI is decomposed into three signals consisting of R, G and B components, and from these signals, Independent Component Analysis (ICA: Independent Component Analysis) A method of extracting a pulse wave signal (PG (plethysmograph) signal) corresponding to a heart beat using

  In addition, in Non-Patent Document 2 described below, when measuring the pulse rate of the subject from the video image of the subject's face, the illumination variation is utilized by utilizing the variation of the background density to prevent the decrease in measurement accuracy due to the illumination variation. The method of excluding the influence of is described.

JP, 2011-130996, A

M.-Z. Poh, D. J. McDuff, and R. W. Picard. "Non-contact, automated cardiac pulse measurements using video imaging and blind source separation." Optics Express, 18 (10): 10762-10774, May 2010. X. Li, J. Chen, G. Zhao, and M. Pietikainen. “Remote heart rate measurement from face videos under realistic situations.” In IEEE Computer Vision and Pattern Recognition (CVPR), pages 4264-4271, June 2014.

A device that measures pulse rate and pulse wave signals from the image of the subject's face can accurately measure the subject's face simply by photographing with the camera without being restricted by background, illumination, face movement, etc. Is desirable.
However, in the techniques described in Patent Document 1 and Non-Patent Documents 1 and 2, the pulse and the like are calculated using averaged data of a large number of pixels included in the ROI set in the face image. It is difficult to obtain measurement results of accuracy.
Also, in the technique of Non-Patent Document 1 using ICA, it has not been confirmed whether the R, G and B component signals generated from the video signal of the ROI satisfy the conditions for applying ICA, so An error may be included, and a reduction in measurement accuracy is a concern.
Moreover, in order to implement the technique of Non-Patent Document 2, it is necessary to make the subject's background uniform and simple, and there is a restriction on the background.

  The present invention has been made in consideration of these circumstances, and even if the subject is in motion, even if the illumination illuminating the subject changes, the pulse rate and pulse wave signal of the subject can be accurately determined from the image of the camera showing the subject It is an object of the present invention to provide a measuring device and a measuring method that can measure the

The present invention is a measuring device that measures periodic fluctuation linked to the heart rate of a subject from an image of the face of the subject, and a pair of small areas consisting of a plurality of pixels from a predetermined area set in the image of the face of the subject. A sample selection unit for selecting a plurality of samples, a sample luminance signal acquisition unit for tracking the position of each sample in each image frame, and acquiring a pair of temporally displaced luminance signals of each sample from pixel values at that position; Assuming that the luminance signal pair of the sample satisfies the applicability of linear BSS (blind source separation), a signal mainly contributed by skin melanin from this luminance signal pair and a signal contributed by blood vessel hemoglobin And a sample pulse wave signal extraction unit that solves a linear BSS problem to estimate and that a signal contributed by hemoglobin is extracted as a pulse wave signal (PG signal); A reliability determination unit that determines the reliability of the extracted PG signal of each sample; a sample data calculation unit that calculates the pulse rate of each sample from the PG signal of each sample determined to be reliable by the reliability determination unit; A sample data grouping unit that sorts the pulse rate of each sample calculated by the sample data calculation unit into groups, and a measurement result of determining the measurement result of the periodic fluctuation of the subject based on the pulse rate of the sample with the largest number of groups. And a determination unit.
In this measuring device, using a pair of small regions selected from a predetermined region (ROI) as a sample, "a signal mainly contributed by melanin in skin" and "a signal contributed by hemoglobin in blood vessels" from a pair of luminance signals of each sample Solve the linear BSS problem to estimate and extract the PG signal. At this time, the problem is solved assuming that the luminance signal pair of the sample satisfies the applicable condition of linear BSS, but the reliability of the obtained PG signal is determined, and the PG signal of the low reliability sample is measured. As a result, compliance with the applicable condition of linear BSS is secured.

Further, in the measuring apparatus of the present invention, it is desirable that the small area be a quadrangle, and one side of the quadrangle be a size of about 10 to 20% of the number of pixels of the face width.
As the small area is larger, the variation of each pixel included therein is averaged to reduce the noise, but a minute variation can not be caught. Also, as the small area is larger, the number of additions for taking the average value increases, so the processing time becomes longer.

Further, in the measuring apparatus of the present invention, the measurement result determination unit determines the pulse rate of the sample with the largest number of samples per group as the pulse rate of the subject, or the number of pulse rates of the samples by group becomes the largest. The PG signals of the samples are averaged to determine the PG signal of the subject.
Thus, this apparatus can accurately measure the subject's pulse rate and PG signal.

Further, in the measurement apparatus of the present invention, it is desirable that the sample selection unit includes a map indicating an effective area in which an effective sample can be obtained in the face, and selects a sample from the effective area.
In this way, unnecessary sample selection can be reduced, and processing time can be shortened.

Further, in the measurement apparatus according to the present invention, when the small area of the sample is constituted of a plurality of pixels, the sample luminance signal acquiring unit averages the pixel values of the plurality of pixels constituting the small area to obtain the luminance signal of the sample. obtain.
Errors in measurement results due to variations in pixel values of pixels can be reduced by averaging pixel values of pixels in a small area.

Further, in the measuring apparatus according to the present invention, the sample pulse wave signal extraction unit calculates an average Euclidean distance between each of two signals obtained by solving the linear BSS problem and the original signal, and the average Euclidean distance is longer. Signal is determined as "a signal contributed by hemoglobin" (= PG signal).
Since melanin is the main factor that determines the color of the skin, the "signal that melanin primarily contributes" is similar to the original signal and the Euclidean distance is close. Compared to that, the "signal contributed by hemoglobin" is far away.

Further, in the measurement device of the present invention, the sample pulse wave signal extraction unit solves the linear BSS problem using the ICA (Independent Component Analysis) method.
As “signals mainly contributed by melanin” and “signals contributed by hemoglobin” are considered to be independent, it is possible to apply the ICA method of decomposing observation data into independent components.

Further, in the measurement device of the present invention, the reliability determination unit calculates the power spectral density of the PG signal of the sample, and sets a ratio r of the power of the maximum peak to the power of the second peak of the second magnitude as a threshold. The comparison is made to determine the reliability of the PG signal. The sample data calculation unit calculates the pulse rate of the sample from the frequency corresponding to the maximum peak of the PG signal determined to be reliable.
From this ratio r, it can be understood how dominant the frequency of the maximum peak in the PG signal is in comparison with other frequencies. The fact that the value r is large is that the PG signal has one main frequency, which indicates that the pulse rate obtained from the frequency is highly reliable.

Further, in the measuring apparatus of the present invention, the sample data grouping unit adds one to the frequency of the class of the histogram corresponding to the pulse rate each time the sample data calculating unit calculates the pulse rate of the sample. The measurement result determination unit determines the pulse rate corresponding to the class of the histogram having the largest frequency as the subject's pulse rate.
Thus, in this device, the subject's pulse rate is determined by majority of sample data.

Further, in the measuring device according to the present invention, the sample data grouping unit generates the pulse rate of each sample calculated by the sample data calculating unit with a value indicating another periodic fluctuation (such as heart rate fluctuation or respiration rate) of the subject. It is also possible to associate and classify, and the measurement result determination unit may be configured to determine the measurement result of the periodic fluctuation interlocking with the subject's heartbeat by averaging the data of the largest cluster.
By doing this, many indicators related to health can be obtained.

Further, the measurement device of the present invention may further include an expression recognition unit that recognizes the expression of the subject from the image of the face of the subject.
By identifying the change in the expression together with the change in the pulse of the subject, it can be determined whether the subject is uplifting in a pleasant state or excited in an unpleasant state.

The present invention is also a measurement method for measuring periodic fluctuation linked to the heart rate of a subject from an image of the face of the subject, which is a small area consisting of a plurality of pixels from a predetermined area set in the image of the face of the subject. Sample selection step of selecting a plurality of pairs as samples, tracking the position of each sample in each image frame, and acquiring a pair of luminance signals of temporally displaced samples of each sample from the pixel value of the position Assuming that the step and the luminance signal pair of the sample satisfy the application condition of linear BSS, “signal mainly contributed by skin melanin” and “signal contributed by blood vessel hemoglobin” from the luminance signal pair A sample pulse wave signal extraction step of solving a linear BSS problem for estimating the "haemoglobin contributed signal" as a pulse wave signal, and a sample pulse wave signal extraction step A reliability determination step of determining the reliability of the pulse wave signal of each sample extracted in step b. Sample data calculation of calculating the pulse rate of each sample from the pulse wave signal of each sample determined to be reliable in the reliability determination step Step, sample data grouping step for sorting the pulse rate of each sample calculated in the sample data calculating step into groups according to group, and the measurement result of the periodic fluctuation of the subject based on the pulse rate of the sample with the largest number by group Determining a measurement result.
In this measurement method, the linear BSS problem is solved on the assumption that the luminance signal pair of the sample satisfies the linear BSS applicability condition, but the reliability of the obtained PG signal of each sample is determined, and the reliability is low. Since the PG signal of the sample is excluded from the calculation of the measurement result, the adaptation to the applicable condition of the linear BSS is secured as a result.

  Further, in the measurement method of the present invention, in the measurement result determination step, the pulse rate of the sample with the largest number in each group is determined as the subject's pulse rate, or the number of pulse rates in the sample by group is the largest. The PG signals of the samples are averaged to determine the PG signal of the subject.

  Further, in the measurement method of the present invention, at the sample pulse wave signal extraction step, the linear BSS problem is solved using the ICA method.

  Further, in the measurement method of the present invention, in the sample pulse wave signal extraction step, the average Euclidean distance between each of the two signals obtained by solving the linear BSS problem and the original signal is calculated, and the average Euclidean distance is longer Signal is determined as "a signal contributed by hemoglobin".

Further, in the measurement method of the present invention, the power spectral density of the PG signal of the sample is calculated in the reliability determination step, and a ratio r of the power of the maximum peak to the power of the second peak of the second magnitude is taken as a threshold. The reliability of the PG signal is determined by comparison, and in the sample data calculation step, the pulse rate of the sample is calculated from the frequency that produces the maximum peak.
The reliability of the PG signal can be evaluated in a short time in a heuristic manner.

Further, in the measurement method of the present invention, each time the pulse rate of the sample is calculated in the sample data calculation step in the sample data grouping step, the frequency of the class of the histogram corresponding to the pulse rate is incremented by 1 and the measurement result In the determination step, the pulse rate corresponding to the class of the histogram with the largest frequency is determined as the subject's pulse rate.
Determine the subject's pulse rate by majority of sample data.

  According to the measurement apparatus and measurement method of the present invention, it is possible to accurately measure a subject's pulse rate and pulse wave signal from an image of a camera that reflects the subject's face even if the subject moves or changes illumination to illuminate the subject. it can.

The figure which shows typically the structure of the measuring device which concerns on embodiment of this invention A diagram showing a pair of image regions ROI and feature points and small regions p1 and p2 of a face image Block diagram showing the configuration of the operation unit of FIG. 1 Flow chart showing the operation procedure of the operation unit of FIG. 3 Diagram showing (a) PG signal and (b) its power spectral density Figure comparing pulse rate measurement accuracy measured by each method Diagram showing the "Communication support robot system" proposed earlier

  FIG. 1 schematically shows a measuring apparatus according to an embodiment of the present invention. This apparatus comprises an imaging device 1 for imaging the face of a subject 4, a computer 2 for measuring a pulse rate and a pulse wave signal (PG signal) from a face image of the subject imaged by the imaging device, and measurement measured by the computer And a display unit 3 for displaying the result. The computer 2 reads the image frame stored in the image recording unit 20 and the image recording unit 20 that stores a plurality of continuous image frames (moving image data) captured by the imaging device 1 and measures the pulse rate and the PG signal And a display control unit 30 for displaying the measurement result calculated by the calculation unit on the display unit 3.

First, the appearance of the skin will be described.
The color of the skin is also influenced by the absorption of light by hemoglobin in the blood, mainly due to the presence of melanin. Thus, the shape of the spectral reflectance curve of the skin is mainly due to the combination of melanin and hemoglobin.
Assuming that the skin is Lambertian (Lambert reflection, completely diffuse reflection), the spectral reflectance Rs (λ) of the skin with respect to the wavelength λ is defined by (Equation 1).

Here, a _m and a _h is a scalar Rm (λ), Rh (λ ) is the spectral reflectance normalized, a _h Rh (lambda) is the contribution of hemoglobin, a _m Rm (λ) is the influence of hemoglobin In the absence of the spectral reflectance, that is, mainly due to the effect of melanin.
Therefore, a pixel value indicating brightness at a point on the skin of a certain camera channel can be modeled by (Equation 2).

Here, _al is a scalar, L (λ) is a spectrum (spectral distribution) of normalized illumination (light source), and C _k (λ) is a spectral response of the channel k of the camera.

Up to this point, it was the appearance of one point on the skin at a certain point in time, but the dimension of time is given as (Equation 3) below in order to consider time change.

Since the production of melanin occurs on a much longer time scale than is considered in videography, it can be assumed that the component of spectral reflectance by melanin is constant. On the other hand, the component for hemoglobin that is affected by the heart beat fluctuates on the video. The spectral response of the camera is of course constant with time.

Now, as shown in FIG. 2, the calculation unit 10 of the computer 2 of FIG. 1 sets a rectangular region of interest (ROI) 41 set in the image of the subject's face to a pair of small regions (pl) , P2) are selected as samples, and the pixel values of multiple samples are used to measure the periodic fluctuation linked to the subject's heartbeat. The position of the sample in the ROI can be set arbitrarily.
In addition, as for a small area | region, it is desirable that it is a quadrangle which makes the magnitude | size of a pixel number about 10 to 20% of the pixel number of the horizontal width of a face one side. As will be described later, since the pixel values of the pixels included in the small area are averaged to obtain the pixel values of the small area, as the small area is larger, the variation of each pixel is averaged and the noise is reduced. Therefore, it is desirable that one side of the small area has a length of 10% or more of the number of pixels in the width of the face. However, if it exceeds 20% of the number of pixels in the width of the face, minute variations become difficult to catch, and the number of additions for taking the average value increases, so the processing time becomes long.

FIG. 3 shows functional blocks that constitute the calculation unit 10. The operation unit 10 sets the ROI in the first image frame to select the sample (small area pair) position, tracks the position in each image frame of the sample, and detects the pixel value of that position. A sample luminance signal acquiring unit 12 for acquiring a pair of temporally displaced luminance signals of each sample from a pair of “signals mainly contributed by skin melanin” and “signals contributed by blood vessel hemoglobin” from the pair of luminance signals Solve the linear BSS problem of “estimating”, and determine the reliability of the extracted PG signal by the sample pulse wave signal extraction unit 13 that extracts the “hemoglobin contributing signal” (= pulse wave signal: PG signal) The reliability determination unit 14, the sample data calculation unit 15 that calculates the pulse rate from the reliable PG signal, and the sample data grouping unit 16 that sorts the pulse rate into groups , By Group number and a measurement result determination unit 17 which determines the pulse rate up to the sample as a pulse rate of the subject.
The sample selection unit 11, the sample luminance signal acquisition unit 12, the sample pulse wave signal extraction unit 13, the reliability determination unit 14, the sample data calculation unit 15, the sample data grouping unit 16 and the measurement result determination unit 17 Is realized by executing processing based on a program.

FIG. 4 shows the processing procedure of the arithmetic unit 10.
The sample selection unit 11 reads out the first image frame from the image frames stored in the image recording unit 20, sets a rectangular ROI (step 1), and randomly selects the first sample (small area) from within the ROI. The pair p1, p2) is selected (step 2).
The sample luminance signal acquiring unit 12 reads all image frames accumulated in the image recording unit 20, and tracks p1 and p2 positions in each image frame.

This tracking is performed as follows.
FIG. 2 shows 66 feature points used for detection and tracking of face feature points in the following paper.
Posefree facial landmark fitting via optimized part mixtures and cascaded deformable shape model. In IEEE International Conference on Computer Vision (ICCV), pages 1944-1951 X. Yu, J. Huang, S. Zhang, W. Yan, and D. Metaxas. , 2013. 4, 7
These 66 feature points can be tracked with respect to the image of the entire image frame.
Here, 17 feature points of the contour portion of the face indicated by the numbers 1 to 17 in FIG. 2 are used for tracking. The shape of the subsequent frame of the ROI set as a rectangle in the first image frame can be obtained by applying rigid body transformation so that the relative position does not change with respect to the tracked feature point. The transformation matrix A of the rigid transformation and the translation vector b are determined by (Equation 4).

Here, vector v _{1, i} is the position of the i-th feature point in frame 1, and vector v _{n, i} is the position of the i-th feature point in the n-th frame. Feature points other than the 17 points are not used because they may indicate non-rigid motion. Equation 4 can be solved by the least squares method.

Coordinate values at the n-th frame of points p1 and p2 set in the ROI of the first frame are determined by the transformation matrix A and the translation vector b.
The sample luminance signal acquiring unit 12 calculates the sample positions in all the frames in this manner, and obtains the pixel values at the positions, thereby representing the signals Ip1 (t), Ip2 (time-variation of the pixel values (luminance) of the samples Get t) (step 3).
Here, in order to reduce the noise due to the variation of the pixel value of each pixel, the pixel values of the pixels included in the small area are averaged to be the luminance of the sample.

The sample pulse wave signal extraction unit 13 aims to obtain a PG signal by applying a linear BSS (Blind Source Separation) to the two signals Ip1 (t) and Ip2 (t) (step 4).
The problem of the linear BSS results in the separation of the signal sources S1, S2 from the equation in which the linear sum of the values of the signal sources S1, S2 becomes the observed values x1, x2, as shown in (Equation 5).

Here, a1, a2, b1, and b2 are constants independent of time. If S1 and S2 are independent of each other, then Eq. 5 can be solved using the ICA method.

The signals Ip1 (t) and Ip2 (t) can be expressed by the following equation 3 from the equation 3.

This signal is not directly applicable to linear BSS because some scalars change time (eg, al (t)). Furthermore, the spectral distribution (L1, L2) of the illumination may be different even at the same time.

In order to apply a linear BSS to this signal, it must be confirmed that Ip1 (t) and Ip2 (t) satisfy the following conditions.
(Condition 1) L ₁ (λ, t) = L ₂ (λ, t) for all times t and all wavelengths λ. That is, although the spectrum of the illumination may change in time, the spectra of the normalized illumination illuminating the two points p1 and p2 must be the same at the same time. In essence, the color of the light source may change with time, but at the same point in time it must be the same color for the two points p1, p2. The relative brightness may be different at the same time.
(Condition 2) Scalars of time t = 1, 2,... N are arranged to define a vector of (Equation 7), and other scalars are similarly defined.

When defined in this way, the normalized vector satisfies the following equation (Equation 8).

In other words, the change in brightness with time between two points may be on different scales, but the relative changes must be equal.

(Condition 3) Similarly, the following equation (equation 9) should be satisfied.

In the case of this measuring apparatus, this condition is satisfied. This is because the blood flow depends on the heartbeat of the heart, but in a normal camera with a frame rate of 30 to 60 Hz, the delay of the PG signal can be neglected even between the most distant points in the face.
(Condition 4) Lighting, hemoglobin and melanin are not uniform over the entire face. That is, to satisfy the relationship of (Equation 10).

In the case of this measuring device, it can be expected that the illumination on the face is not uniform. Also, the flow of blood is different in each part of the face.

Although it can not be expected that all pairs of two points selected for the sample satisfy all the above conditions, the sample pulse wave signal extraction unit 13 performs processing assuming that the point satisfying these conditions is found for the time being. to continue.
Assuming that these conditions are satisfied, Ip1 (t) and Ip2 (t) are expressed by the following equation (Equation 11).

Here, cl (t) is the t-th element of the vector cl defined by (Equation 8), and ch (t) is the t-th element of the vector ch defined by (Equation 9).

In order to simplify (Equation 11), the following equation (Equation 12) is substituted.

The same replacement is performed for Ip2 (t).
Then, equation (11) can be expressed as the following equation (equation 13).

Further, from condition 4, there is a relation of (Equation 14).

Assuming that the elements of Im and Ih are independent in (Equation 13), (Equation 13) can be solved using FastICA, which is a standard algorithm of independent component analysis, and Im and Ih can be Can be asked aside).
The sample pulse wave signal extraction unit 13 calculates which of the obtained Im and Ih is the PG signal, and which one is closer to the average Euclidean distance with the original signal (Ip1, Ip2). The person is determined to be Im. This is because melanin is the main factor that determines the skin color and is close to the original signal.
Another remaining signal Ih is a PG signal (step 4).

The reliability determination unit 14 converts non-stationary tendency and noise from the obtained PG signal into a frequency domain in order to determine the reliability of the signal after using a time filter and a moving average filter. And calculate the power spectral density (PSD) (step 5).
The PSD of the PG signal shown in FIG. 5A is shown in FIG. 5B. Here, the frequency range of the PSD is set to 0.7 to 4 Hz in consideration of a normal human heart rate.
The reliability determination unit 14 calculates a ratio V1 / V2 (= r) between the power V1 of the maximum peak of the power spectrum density and the power V2 of the second peak of the second magnitude (step 6), and the ratio r Is compared to a threshold (eg, 2) (step 7).
This ratio r is a guide for identifying how dominant the frequency of the maximum peak is in the PG signal compared to other frequencies. The large value of this value means that the PG signal has one main frequency, which indicates that the pulse rate obtained from the frequency is highly reliable.
If the ratio r is less than the threshold (No in step 7), the process returns to step 2 to select another sample (small area pair p1, p2) from within the ROI of the first image frame, repeat.

If the ratio r is larger than the threshold (Yes in step 7), the reliability determination unit 14 determines that the PG signal extracted in step 4 is reliable. The sample data calculation unit 15 multiplies the frequency of the maximum peak of the reliable PG signal by 60 to calculate the pulse rate per minute (step 8).
Each time the sample data calculation unit 15 calculates the pulse rate of the sample, the sample data grouping unit 16 adds one to the frequency of the class of the histogram corresponding to the pulse rate (step 9).
If the number of pulse rates of the sample calculated by the sample data calculation unit 15 does not reach the specified number (No in step 10), the process returns to step 2 and another sample (small area The pair p1, p2) is selected, and the processing of step 3 and subsequent steps is repeated.
In step 10, when the calculated number of pulse rates of the sample reaches the specified number (Yes in step 10), the measurement result determination unit 17 compares the frequencies of each class of the histogram, and based on majority decision, the maximum frequency. The pulse rate classified into the following ranks is determined as the subject's pulse rate (step 11).
In step 10, the number of pairs of small areas selected in step 2 is counted, and the processes of steps 2 to 9 are repeated until the number reaches a predetermined number, and the number (repetition number) reaches a predetermined number. If it does, or if the number of reliable pulses has reached the specified number before that, step 11 may be performed.
The pulse rate determined by the measurement result determination unit 17 is output to the display unit 3 and displayed (step 12).

As described above, in this measuring device, the pair of small regions selected from the ROI is used as a sample, and the PG signal is calculated using the pixel values of many samples. Therefore, higher measurement accuracy can be obtained as compared to the conventional method of calculating the pulse and the like based on the average pixel value of the ROI.
Moreover, in this measuring device, calculation of the PG signal is performed on the assumption that the pixel value of each sample satisfies the application condition of linear BSS, but the reliability of the PG signal of the obtained sample is judged to determine the reliability. Since the PG signal of the subject is determined excluding the PG signal of the low sample, the adaptation to the application conditions of the linear BSS is consequently secured, and high measurement accuracy is obtained.

FIG. 6 shows the result of measurement of the pulse according to the present invention in comparison with the result of measurement according to the other method.
In FIG. 6, the results obtained by each method of FIG. 6 are compared with 487 examples of published databases (MANHOB-HCI Database).
In FIG. 6, in the “all small area data use” method, the ROI is divided into small areas (one size is 41 × 41 pixels) so that there is no overlap, and the average time change data of the pixels in each small area The ICA is applied to the entire data of all the small regions to calculate two signals, and the pulse is calculated from one of the PG signals. That is, the difference between the present invention and the present invention is that calculation is not performed on pairs of small regions, and the selection of small regions having high reliability is not performed in calculating the pulse rate.
Also, “Poh 2011 [1]” is a method described in Non-Patent Document 1 and “Li 2014 [2]” is a method described in Non-Patent Document 2.
In the present invention, the size of the small area is set to 41 × 41 pixels, the number of selection (the number of repetitions) of the small area pair is set to 500, and the threshold of the reliability ratio r determination is set to 2. The 41 pixels on one side of the small area correspond to 13.7% of the number of pixels 300 in the width of the face.
It can be seen from FIG. 6 that when the method of the present invention is used, the pulse rate is obtained with high accuracy.
In the present invention, the reliability of the PG signal is determined from the ratio of the maximum peak to the second peak of the power spectral density of the PG signal, and the final pulse measurement value is determined by majority determination of the sample pulse rate. ing. FIG. 6 shows that the measurement accuracy does not decrease even if the measurement time is shortened using such a heuristic method.

The relationship between the pair of small areas set as a sample in the present invention and the application conditions (conditions 1 to 4) of the linear BSS can be considered as follows.
Although not all point pairs satisfy the conditions 1 to 4, it is considered that a sufficient number of point pairs satisfy (or almost satisfy) these conditions.
Conditions 1 and 2 require that the two points of the pair be illuminated by the same light source at each point in time (although the illumination intensity for the two points may be different). While this may seem to be a limiting condition, it can be expected that many camera images will have many points that satisfy this. There are many light sources in the real environment, but it is thought that the same light source affects the whole part of the face.
In other words, it is impossible for most points to be illuminated by completely different light sources.
Further, in the condition 1 and the condition 2, if the light source is the same, the spectrum may change in time in any way, so there is flexibility. For example, a computer monitor (display) is one light source in which the spectrum of the light source fluctuates.
As described above, conditions 3 and 4 are considered to satisfy the pair of small areas set in the present invention.
However, finding a point that satisfies the conditions 1 to 4 is not easy. Many factors are related to whether these conditions are satisfied.
Therefore, in the present invention, as described above, the PG signal is calculated on the assumption that the pixel value of each sample satisfies the application condition of linear BSS, and among the obtained PG signals, those with low reliability are excluded. I have taken the method. By doing this, it is possible to avoid the difficult judgment of which pair satisfies the condition.

Although the case where the sample is randomly selected from the ROI is described here, by repeating the practice of the present invention, the position of the sample from which a highly reliable PG signal can be obtained can be grasped, and such a sample position You can create a map that includes If this map is held and a sample is selected, unnecessary calculations are reduced and processing can be speeded up.
Moreover, although measurement of a subject's pulse rate was explained here, a subject's pulse wave signal (PG signal) can also be measured by the device of the present invention. In this case, PG signals of the samples classified into the histogram class having the largest frequency are averaged to obtain PG signals of the subject.
Also, here, the case is shown where the PG signals of the samples are sorted using a histogram, but the pulse and PG signals of each sample are correlated with the values of the subject's heart rate fluctuations and respiration rate to obtain K-means and hierarchical clustering. The classification may be performed using the above, and data of maximum clusters may be averaged to obtain biological information of the subject. Also, biological information may be obtained using a method called RANSAC (random sample condition).

The measurement apparatus of the present invention is a "communication support robot system" (Japanese Patent Laid-Open No. 2015-184597) previously proposed by the present inventors etc., that is, supports communication between a single elderly person and a close relative using a robot. It can also be used for systems.
In this system, as shown in FIG. 7, a robot 37, a camera 33 for realizing a video telephone function, a speaker 361, a microphone 362, a display 35, and a computer 31 are disposed on the side of an elderly person at risk of dementia. An image of an elderly person photographed by the camera 33 and a voice of the elderly person collected by the microphone 362 can be displayed on the display 45 and the speaker 461 of the videophone on the close relative side.
The close relatives communicate directly with the elderly through a video phone in the "direct response mode", and the robot 37 involuntarily communicates with the elderly about the relatives' intention through the robot 37. You can select the "robot support mode" that you want to interact with.

Introducing the device of the present invention into this system, continuously measuring the pulse rate of elderly people who interact with relatives and robots 37, and examining their fluctuations is effective in understanding changes in the elderly's emotions. It is. It is well known that when there is a change in emotion, a change in pulse appears.
However, I do not know if the emotion at that time is pleasant or unpleasant.
Therefore, by combining with facial expression recognition means that recognizes the facial expression of the elderly from the face image of the elderly, when there is a pulse fluctuation and an expression representing pleasure, we prefer the topic of dialogue and respond positively It can be judged. Also, if there is no change in pulse rate, there may be a problem with the emotional situation. Thus, the magnitude of presence or absence of the pulse rate fluctuation at the time of dialogue is one of the indices to know the degree of dementia.

  The measurement apparatus and measurement method of the present invention can accurately measure the pulse rate, pulse wave signal, etc. of the subject from the video image of the subject even if the face of the subject moves or the illumination striking the subject changes. It can be used in a wide range of fields, such as medical facilities, nursing facilities, and training facilities.

Reference Signs List 1 imaging device 2 computer 3 display unit 4 subject 10 arithmetic unit 11 sample selection unit 12 sample luminance signal acquisition unit 13 sample pulse wave signal extraction unit 14 reliability determination unit 15 sample data calculation unit 16 sample data grouping unit 17 determination of measurement result Part 20 Image recording part 30 Display control part 41 Region of interest (ROI)

Claims (21)

A measuring device for measuring periodic fluctuation linked to a subject's heartbeat from an image of the subject's face,
A sample selection unit which selects a plurality of pairs of small areas consisting of a plurality of pixels from a predetermined area set in the image of the face of the subject as a sample;
A sample luminance signal acquisition unit for tracking the position of each sample in each image frame and acquiring a pair of temporally displaced luminance signals of each sample from the pixel value of the position;
Assuming that the luminance signal pair of the sample satisfies the applicability of linear BSS (blind source separation), a signal mainly contributed by skin melanin from the luminance signal pair and a signal contributed by blood vessel hemoglobin And a sample pulse wave signal extraction unit that solves a linear BSS problem of estimating the H.sub.ss and extracts the signal contributed by the hemoglobin as a pulse wave signal;
A reliability determination unit that determines the reliability of the pulse wave signal of each sample extracted by the sample pulse wave signal extraction unit;
A sample data calculation unit that calculates the pulse rate of each sample from the pulse wave signal of each sample determined to be reliable by the reliability determination unit;
A sample data grouping unit that sorts the pulse rate of each sample calculated by the sample data calculation unit into groups;
A measurement result determination unit that determines the measurement result of the periodic fluctuation of the subject based on the pulse rate of the sample having the largest number for each group;
Measuring device characterized by having.   The measuring device according to claim 1, wherein the small area is a quadrangle, and one side of the quadrangle has a size of about 10 to 20% of the number of pixels of the width of the face. Measuring device.   The measuring apparatus according to claim 1, wherein the measurement result determining unit determines the pulse rate of the sample having the largest number for each group as the pulse rate of the subject.   The measurement apparatus according to claim 1, wherein the measurement result determination unit averages the pulse wave signal of the sample at which the number of pulse rates of the group-specific samples is maximized and the pulse wave of the subject. Measuring apparatus characterized by determining as a signal.   The measurement apparatus according to claim 1, wherein the sample selection unit has a map indicating an effective area in which an effective sample can be obtained in the face, and selecting the sample from the effective area. Measuring device to be characterized.   The measurement apparatus according to claim 1, wherein the sample luminance signal acquisition unit obtains the luminance signal of the sample by averaging pixel values of a plurality of pixels forming the small area. .   The measurement apparatus according to claim 1, wherein the sample pulse wave signal extraction unit solves the linear BSS problem using an ICA (Independent Component Analysis) method.   The measurement apparatus according to claim 1, wherein the sample pulse wave signal extraction unit calculates an average Euclidean distance between each of the two signals obtained by solving the linear BSS problem and an original signal, A measuring apparatus characterized in that a signal having a longer average Euclidean distance is determined as a signal to which the hemoglobin contributes.   The measurement apparatus according to claim 1, wherein the reliability determination unit calculates a power spectral density of the pulse wave signal of the sample, and the power of the maximum peak and the power of the second peak of the second magnitude. And measuring the reliability of the pulse wave signal by comparing the ratio of.   The measuring apparatus according to claim 9, wherein the sample data calculating unit calculates the pulse rate of the sample from the frequency at which the maximum peak is obtained.   The measuring apparatus according to claim 1, wherein the sample data grouping unit is configured to set the frequency of the class of the histogram corresponding to the pulse rate to 1 each time the sample data calculating unit calculates the pulse rate of the sample. A measuring device characterized by adding.   The measuring apparatus according to claim 11, wherein the measurement result determining unit determines a pulse rate corresponding to a class of a histogram having the largest frequency as the pulse rate of the subject.   The measuring apparatus according to claim 1, wherein the sample data grouping unit classifies the pulse rate of each sample calculated by the sample data calculating unit in association with a value indicating another periodic fluctuation of the subject. The measurement apparatus according to claim 1, wherein the measurement result determination unit determines the measurement result of periodic fluctuation interlocking with the heart rate of the subject by averaging data of the largest cluster.   The measurement apparatus according to any one of claims 1 to 13, further comprising: an expression recognition unit that recognizes an expression of the subject from the image of the face of the subject. A measurement method for measuring periodic fluctuation linked to a subject's heartbeat from an image of the subject's face, comprising:
A sample selection step of selecting a plurality of pairs of small areas consisting of a plurality of pixels from a predetermined area set in the image of the face of the subject as a sample;
A sample luminance signal acquiring step of tracking a position of each of the samples in each image frame and acquiring a pair of temporally displaced luminance signals of each sample from pixel values of the position;
Assuming that the luminance signal pair of the sample satisfies the applicability of linear BSS (blind source separation), a signal mainly contributed by skin melanin from the luminance signal pair and a signal contributed by blood vessel hemoglobin And a sample pulse wave signal extraction step of solving the linear BSS problem of estimating the phase and extracting the signal contributed by the hemoglobin as a pulse wave signal;
A reliability determination step of determining the reliability of the pulse wave signal of each sample extracted in the sample pulse wave signal extraction step;
A sample data calculation step of calculating a pulse rate of each sample from pulse wave signals of each sample determined to be reliable in the reliability determination step;
A sample data grouping step of sorting the pulse rate of each sample calculated in the sample data calculation step into groups;
Determining a measurement result of the periodic fluctuation of the subject based on the pulse rate of the sample having the largest number in each group;
A measuring method comprising:   The measurement method according to claim 15, wherein in the measurement result determination step, the pulse rate of the sample having the largest number in each group is determined as the pulse rate of the subject.   The measurement method according to claim 15, wherein in the measurement result determination step, the pulse wave of the subject is averaged by averaging the pulse wave signal of the sample at which the number of pulse rates of the samples in each group is maximized. Measuring method characterized by determining as a signal.   The measurement method according to claim 15, wherein in the sample pulse wave signal extraction step, the linear BSS problem is solved using an ICA (independent component analysis) method.   The measurement method according to claim 15, wherein the sample pulse wave signal extraction step calculates an average Euclidean distance between each of the two signals obtained by solving the linear BSS problem and the original signal, A measurement method characterized in that a signal with a longer average Euclidean distance is determined as a signal to which the hemoglobin contributes.   The measurement method according to claim 15, wherein the power spectrum density of the pulse wave signal of the sample is calculated in the reliability determination step, and the power of the maximum peak and the power of the second peak of the second magnitude. And determining the reliability of the pulse wave signal by comparing the ratio of the two with a threshold value, and calculating the pulse rate of the sample from the frequency at which the maximum peak occurs in the sample data calculation step.   The measurement method according to claim 15, wherein in the sample data grouping step, every time the pulse rate of the sample is calculated in the sample data calculating step, the frequency of the class of the histogram corresponding to the pulse rate is 1 Adding, and in the measurement result determination step, determining a pulse rate corresponding to a class of a histogram having the largest frequency as the pulse rate of the subject.