JP6928219B2 - Pulse cycle detection device, pulse cycle detection method, pulse cycle detection program, and pulse wave detection device - Google Patents

Description

The present invention relates to a pulse cycle detection device, a pulse cycle detection method, a pulse cycle detection program, and a pulse wave detection device.

Conventionally, a device for non-contactly detecting a person's pulse cycle from a face image taken by a camera or the like is widely known. For example, the apparatus described in Patent Document 1 applies independent component analysis to time-series data of luminance values for each light wavelength component of a face image to obtain spectral distributions of a plurality of independent signals. Further, the spectrum distribution of the independent signal is subjected to frequency analysis using FFT (Fast Fourier Transform) to obtain the maximum value of the spectrum. Then, the pulse rate (corresponding to the reciprocal of the pulse period) is detected by performing a weighting calculation with a plurality of maximum values of the spectrum as inputs.

Japanese Unexamined Patent Publication No. 2012-239661

The device described in Patent Document 1 acquires time-series data over a plurality of pulses, and can be detected more accurately as the number of acquired pulses increases. Therefore, if the pulse cycle is to be detected with high accuracy, the delay until the detection of the pulse rate is completed becomes large. Therefore, it is not suitable for applications such as detecting an instantaneous pulse rate from a facial image or the like in almost real time. For example, in a scene where the instantaneous heart rate changes suddenly with a change in emotion, the pulse cycle or the instantaneous pulse rate corresponding to the instantaneous heart rate is detected in real time.

The present invention has been made in view of such circumstances, and an object of the present invention is a pulse cycle detection device for shortening the pulse cycle detection time in order to obtain an instantaneous pulse rate, a pulse cycle detection method, and a pulse. It is an object of the present invention to provide a periodic detection program and a pulse wave detection device.

In order to solve the above problems, the present invention is a pulse cycle detection device that detects the pulse cycle of a person from a time series image, and has a pulse that detects time series data related to a luminance value for a plurality of unit regions in the image. Statistics for the wave detection unit, the unit cycle calculation unit that calculates the unit period that is the pulse period for each unit region based on the time series data related to the brightness value, and the plurality of unit periods calculated for different unit regions. It is characterized by including a statistical processing unit that performs processing and determines the pulse cycle of the person.

Further, in order to solve the above problems, the present invention is a pulse cycle detection method for detecting a person's pulse cycle from a time series image, and detects time series data related to a luminance value for a plurality of unit regions in the image. For the pulse wave detection process to be performed, the unit cycle calculation process for calculating the unit cycle which is the pulse cycle for each unit region based on the time series data related to the brightness value, and the plurality of unit cycles calculated for different unit regions. It is characterized by including statistical processing for determining the pulse cycle of the person by performing statistical processing.

Further, in order to solve the above-mentioned problems, the present invention uses a computer to perform pulse wave detection processing for detecting time-series data related to brightness values for a plurality of unit regions in a time-series image, and time-series data related to the brightness values. Based on this, a unit cycle calculation process for calculating a unit cycle, which is a pulse cycle for each unit region, and a statistical process for determining a person's pulse cycle by performing statistical processing on a plurality of the unit cycles calculated for different unit regions. And, it is characterized by executing.

According to the above configuration, the pulse cycle can be detected accurately even if the detection time is shortened by increasing the number of detection points instead of acquiring the time series data over a large number of pulse times. Specifically, even if the calculation results of the unit cycle in some unit regions vary, the detection time can be shortened by performing statistical processing on the calculation results of a plurality of unit cycles calculated for different unit regions. However, it is possible to accurately detect the pulse cycle of a person.

Further, according to the above configuration, not only when the detection time is shortened, but also when the detection accuracy in each unit region is low due to other causes, the pulse cycle of the person can be detected with high accuracy.

Further, in the present invention, the pulse wave detection unit is characterized in that a plurality of the unit regions are set in the skin region of the person.

According to the above configuration, it is possible to reduce the noise included in the calculation result of the unit cycle for each unit region even if the skin portion of the person is included in the image.

Further, the present invention is characterized in that, in the above invention, the pulse wave detection unit detects the relative ratio of the luminance values of different light wavelength components as the time series data.

According to the above configuration, it is possible to detect the pulse cycle with high accuracy.

Further, the present invention is characterized in that, in the above invention, the unit cycle calculation unit calculates the latest cycle based on the time series data as the unit cycle.

According to the above configuration, since the unit cycle can be calculated if the time series data is for one cycle, the pulse cycle can be detected in a short time.

Further, in the present invention, in the above invention, the unit cycle calculation unit detects a peak point at which the time series data has a maximum value or a minimum value, and calculates the time difference between adjacent peak points as the unit period. It is characterized by.

According to the above configuration, the unit cycle data for each unit region is calculated when the peak point for one cycle is detected from the time series data of the brightness value for each unit region. Therefore, by performing statistical processing on the unit cycle for each unit region, it is possible to detect the pulse cycle of a person in a short time.

Further, in the present invention, in the above invention, the unit period calculation unit converts the time series data by the combined product of the time series data and the reference function, and based on the converted time series data, the unit area. It is characterized in that the unit cycle is calculated for each unit.

According to the above configuration, the error component of the time series data regarding the luminance value is reduced by the combined product with the reference function. Then, the extreme value of the brightness value is obtained based on the time series data in which the error component is reduced. This makes it possible to improve the detection accuracy of the pulse cycle.

Further, the present invention is characterized in that, in the above invention, the statistical processing unit determines the mode, average or median of a plurality of the unit cycles calculated for different unit regions as the pulse cycle of the person. do.

According to the above configuration, the unit period for each unit area is statistically processed over the entire analysis range. As a result, the influence of the phase difference due to the blood flow velocity in each unit region can be suppressed, and the detection accuracy of the pulse cycle can be improved.

Further, in the present invention, in the above invention, the statistical processing unit assumes that the distribution of a plurality of the unit regions calculated for different unit regions is a normal distribution, and determines the pulse period of the person based on the assumed normal distribution. It is characterized by determining.

According to the above configuration, the resolution of the pulse period is enhanced by statistically increasing the resolution of the distribution of the plurality of unit regions. This makes it possible to detect the pulse period with the same resolution as that of a high frame rate camera even when a low frame rate camera is used.

Further, in the present invention, in the above invention, the statistical processing unit calculates the reliability of the unit cycle for each unit region based on the time series data, and for each unit region according to the calculated reliability. The unit cycle is weighted to determine the pulse cycle of the person.

According to the above configuration, the pulse cycle can be detected more robustly by lowering the weighting of the unit region other than the skin portion and increasing the weighting of the unit region of the skin portion.

Further, in the present invention, the pulse rate is calculated based on the pulse cycle determined by the statistical processing unit in the above invention.

According to the above configuration, it is possible to detect the pulse rate of a person in a short time.

Further, in order to solve the above problems, the present invention is a pulse wave detection device that detects a pulse wave of a person from a time series image, and has a luminance value for each different light wavelength component for a plurality of unit regions in the image. It is characterized by including a pulse wave detection unit that detects the relative ratio of the above as time series data.

According to the above configuration, based on the relative ratio of the luminance values of the plurality of light wavelength components, it is possible to detect the pulse wave of a person with high accuracy while suppressing the influence of ambient light.

Further, in the present invention, in the above invention, a unit for calculating a unit cycle, which is a pulse cycle for each unit region, based on the pulse wave detection device having the above configuration and the time series data detected by the pulse wave detection device. A cycle calculation unit is provided, and the pulse cycle of the person is determined based on the unit cycle calculated by the unit cycle calculation unit.

According to the above configuration, based on the relative ratio of the luminance values of the plurality of light wavelength components, the pulse period of the person is analyzed while suppressing the influence of ambient light. This makes it possible to detect the pulse cycle in a short time with high accuracy.

According to the present invention, it is possible to shorten the detection time of the pulse cycle of a person.

The block diagram which shows the functional structure of the 1st Embodiment of a pulse cycle detection apparatus. A graph showing the correlation between the relative ratio of green luminance values and pulse pressure. A graph showing the characteristics of the first-order differential Gaussian function. A graph showing the characteristics of pulse waves before and after conversion. The flowchart which shows the processing content of the pulse cycle detection processing executed by the pulse cycle detection apparatus of the same embodiment. (A) is a schematic diagram showing an example of a person's attributes, and (b) is a graph showing an example of a frequency distribution of unit period data obtained from the person shown in FIG. 6 (a). (A) is a schematic diagram showing an example of the attributes of a person, and (b) is a graph showing an example of the frequency distribution of unit period data obtained from the person shown in FIG. 7 (a). (A) is a schematic diagram showing an example of the attributes of a person, and (b) is a graph showing an example of the frequency distribution of unit period data obtained from the person shown in FIG. 8 (a). (A) is a schematic diagram showing an example of an analysis range for a facial region, and (b) is a graph showing an example of a frequency distribution of unit period data obtained from the analysis range shown in FIG. 9 (a). (A) is a schematic diagram showing an example of an analysis range for a facial region, and (b) is a graph showing an example of a frequency distribution of unit period data obtained from the analysis range shown in FIG. 10 (a). (A) is a schematic diagram showing an example of an analysis range for a facial region, and (b) is a graph showing an example of a frequency distribution of unit period data obtained from the analysis range shown in FIG. 11 (a). A graph for explaining an example of an analysis method for the frequency distribution of unit period data. A graph for explaining an example of a pulse wave analysis method.

The pulse cycle detection device, the pulse cycle detection method, and the pulse cycle detection program of the present invention detect a pulse wave from a change in the brightness value of an image of a person's skin area, and detect the pulse cycle. By statistically processing the period of the pulse wave detected at the above location, accurate detection can be achieved in a short time.

It is known that blood vessels near the surface of the skin are also beating in response to the heartbeat, which is the beating of the heart, and that pulse waves near the surface of the skin can be detected from changes in the brightness value of the image of the skin area. Has been done. Since it takes time for blood pressure fluctuations to be transmitted from the heart to the surface of the skin, the phases of the heartbeat and pulse do not always match, but the cycles usually do. Therefore, the heartbeat cycle can be estimated by detecting the pulse wave and obtaining the cycle.

Even on the surface of the skin, the phases of the pulses do not always match in different places. Therefore, the pulse can be detected more accurately by detecting the pulse from the change of the brightness value in the narrow region than by detecting the pulse from the average value of the brightness value in the wide region. When detected in a narrow area, an abnormal area such as a part with poor blood flow or a part other than the skin becomes a detection area, and it may be difficult to detect the pulse, but it was detected normally by detecting in a large number of areas. Many areas can be included.

The pulse cycle detected in the normal region is a certain range of cycles corresponding to the heartbeat cycle, while the pulse cycle detected in the abnormal region is concentrated in a specific cycle different from the heartbeat cycle. There is no. Therefore, the pulse cycle can be detected with high accuracy by statistically processing the pulse cycle detected in a large number of regions.

As described above, since the accuracy is improved by detecting the pulse in a large number of regions, it is possible to prioritize the detection in a short time rather than the accurate detection in each region. In the present invention, the time from the peak of the pulse wave to the next peak is calculated corresponding to the calculation method of RRI (R-R interval). Since it can be calculated if time-series data capable of detecting one pulse can be obtained, the pulse can be detected in a short time. If the period of one pulse can be calculated, a calculation method other than the time between peaks can be used.

(First Embodiment)
Hereinafter, the first embodiment of the pulse cycle detection device will be described with reference to the drawings.

The pulse cycle detection device of the present embodiment detects a facial region as a skin region from a person in an image, and acquires luminance values of a plurality of color components (light wavelength components) for the facial region. In addition, the pulse wave of a person is detected based on the relative ratio of the brightness values of each color. In this respect, the pulse detection device also functions as a pulse wave detection device. Further, the pulse cycle detection device calculates a unit cycle, which is a pulse cycle for each pixel (unit region), based on the pulse wave of a person. Then, by performing statistical processing on the unit cycle for each pixel, the representative value of the unit cycle for each pixel obtained from the face region of the person is determined as the pulse cycle of the person.

Specifically, as shown in FIG. 1, the pulse cycle detection device 100 includes a control unit 200 that controls the pulse cycle detection process and a pulse cycle detection program that the control unit 200 executes during the pulse cycle detection process. It is a computer having a storage unit 300 for storing various programs including the above and various data read / written by the control unit 200 when the program is executed. Then, the control unit 200 executes the pulse cycle detection program stored in the storage unit 300 to execute the face detection unit 210, the brightness value acquisition unit 220, the pulse wave detection unit 230, the unit cycle calculation unit 240, and the statistical processing unit. It functions as 250 and an output unit 260.

The face detection unit 210 acquires an image frame from a moving image (time-series image) taken by the camera 10. Further, the face detection unit 210 detects a face region from the image frame acquired in this way by template matching or the like, and outputs the data of the detected face region to the brightness value acquisition unit 220.

The range of the face area can be set arbitrarily, but as an example, the upper end can be 60 pixels above the eyebrows, and the left and right lower ends can be the contour of the face. At this time, assuming that the size of the face region is 400 pixels in width and 500 pixels in height, the luminance values are acquired for 200,000 pixels, and the following processing is performed.

The brightness value acquisition unit 220 acquires brightness values of three colors of red, green, and blue as data for each pixel included in the face area of a person based on the data of the face area input from the face detection unit 210. .. In this case, the luminance value acquisition unit 220 follows the movement of a person in the moving image and associates the pixels between continuous image frames with each other. As a result, the luminance value acquisition unit 220 acquires the luminance value of each color for each pixel as time-series waveform data (time-series data). Then, the luminance value acquisition unit 220 outputs the luminance value of each color acquired in this way to the pulse wave detection unit 230.

The pulse wave detection unit 230 smoothes the brightness value of each color by inputting the brightness value of each color input from the brightness value acquisition unit 220 into the image filter. The image filter is a filter for averaging and smoothing the luminance values of each color in a local region such as a square region including a target pixel and surrounding pixels.

Further, the pulse wave detection unit 230 calculates the relative ratio of the brightness values of each color after smoothing as the pulse wave detection process. Specifically, the pulse wave detection unit 230 takes the red luminance value R, the green luminance value G, and the blue luminance value B as inputs, and sets the relative ratio Gr (= G / (R + G + B)) of the green luminance value. calculate. In this case, the relative ratio Gr of the green luminance value reflects the time change of the pulse of the person, and the tendency is maintained even if the ambient light is incident on the moving image. Therefore, the pulse wave detection unit 230 detects the time-series waveform data 310 for the relative ratio Gr of the green luminance value as a person's pulse wave.

FIG. 2 shows time-series waveform data 310 for the relative ratio Gr of green brightness values included in the facial region of a person in a moving image, and a pulse electrically detected from the person using a pulse sensor when the moving image is taken. It is shown in comparison with the time series waveform data for pressure. As shown in the figure, in the process of changing the pulse pressure periodically, while the relative ratio Gr of the green luminance value decreases while the pulse pressure increases, while the pulse pressure decreases. The relative ratio Gr of the green brightness value is increasing. That is, there is a negative correlation between the relative ratio Gr of the green luminance value and the pulse pressure. This is considered to be due to the absorption characteristics of hemoglobin in blood of visible light. Specifically, the absorption characteristic of hemoglobin in blood has a peak in the wavelength band of 520 to 600 nm, and overlaps with the wavelength band in which the translucency of the green filter of the camera 10 becomes high. Therefore, when the blood flow to the capillaries on the face surface increases, the relative amount of hemoglobin in the blood also increases, so that the absorbance of green light on the face surface increases. Since the camera 10 receives the light reflected on the face surface, the relative ratio Gr of the green luminance value in the moving image decreases when the relative amount of hemoglobin in the blood increases. On the contrary, when the relative amount of hemoglobin in blood decreases, the amount of green absorption on the face surface decreases, so that the relative ratio Gr of the green luminance value in the moving image increases.

As shown in FIG. 1, the unit period calculation unit 240 is a composite product of the time-series waveform data 310 of the pulse wave of the person detected by the pulse wave detection unit 230 and the reference function data 320 stored in the storage unit 300. Convolution integral) is calculated. In the present embodiment, the unit period calculation unit 240 uses the first-order differential Gaussian function shown by the following [Equation 1] as an example of the reference function data 320.

FIG. 3 is a graph in which the first-order differential Gaussian function is plotted on the coordinate plane. As shown in the figure, this function is a function that shows a waveform with one output value when the variable t is on the horizontal axis. In the example shown in the figure, the window width of the function is set to "25". It is set. The unit period calculation unit 240 defines 25 points plotted corresponding to the window width in this function as discrete data. Further, the unit period calculation unit 240 performs the discrete value data of the function and the time-series waveform on condition that the number of data of the time-series waveform data 310 reaches a predetermined number of data (25) corresponding to the window width of the function. The combined product with the data in the latest section corresponding to the window width of the function in the data 310 is calculated. Then, the unit cycle calculation unit 240 stores the calculation result as the differential value data 330 (see FIG. 1) in the storage unit 300.

In FIG. 4, the pulse wave of the person before conversion is shown by a broken line, and the pulse wave of the person after conversion is shown by a solid line. As shown in the figure, the pulse wave of the person before conversion has a waveform that fluctuates finely with the passage of time, while the pulse wave of the person after conversion has a smooth fluctuation with the passage of time. It has a waveform. Further, the zero cross point at which the output value becomes zero in the pulse wave of the person after conversion corresponds to the peak point at which the output value becomes the maximum value in the pulse wave of the person before conversion. Then, the unit period calculation unit 240 monitors the differential value data 330, and when the differential value data 330 is inverted from positive to negative, the current time and the zero cross point data 340 immediately before being stored in the storage unit 300. Calculate the difference with. Further, the unit cycle calculation unit 240 calculates the calculated difference as the unit cycle data 350 which is the pulse cycle for each pixel as the unit cycle calculation process, and stores the calculated unit cycle data 350 in the storage unit 300. Further, the unit cycle calculation unit 240 updates the current time as zero cross point data 340 stored in the storage unit 300. After that, the unit cycle calculation unit 240 updates the unit cycle data 350 stored in the storage unit 300 every time a zero cross point appears.

As shown in FIG. 1, the statistical processing unit 250 determines the pulse period of a person by performing statistical processing on the unit cycle data 350 for each pixel calculated by the unit cycle calculation unit 240 as statistical processing. .. In the present embodiment, the statistical processing unit 250 reads the unit period data 350 of each pixel in the face region from the storage unit 300. Then, the statistical processing unit 250 obtains the frequency distribution of the appearance frequency of the unit cycle data 350 for each pixel on the condition that the ratio of pixels different from the initial value of the unit cycle data 350 is a predetermined ratio or more. Then, the statistical processing unit 250 determines the unit cycle data 350, which is the mode, as the pulse cycle (RRI: R-R interval) of the person, and outputs the determined pulse cycle by the output unit 260.

Next, the specific processing contents of the pulse cycle detection process executed by the pulse cycle detection device 100 of the present embodiment will be described.

As shown in FIG. 5, in the pulse cycle detection process, the face detection unit 210 first acquires an image frame of a moving image taken by the camera 10 (step S10). Then, the face detection unit 210 detects the face region from the image frame acquired in the previous step S10 (step S11).

The brightness value acquisition unit 220 acquires the brightness value of each color for each pixel of the face region detected by the face detection unit 210. The pulse wave detection unit 230 smoothes the brightness value of each color by inputting the brightness value of each color acquired by the brightness value acquisition unit 220 into the image filter (step S12).

Further, the pulse wave detection unit 230 detects the time-series waveform data 310 for the relative ratio Gr of the green brightness value of each pixel as a person's pulse wave based on the brightness value of each color smoothed in the previous step S12. (Step S13). Further, the pulse wave detection unit 230 stores the detected time-series waveform data 310 in the storage unit 300.

Then, when the number of data of the time-series waveform data 310 stored in the storage unit 300 reaches a predetermined number of data (step S14 = YES), the unit cycle calculation unit 240 refers to the time-series waveform data 310. The combined product with the first-order differential Gaussian function as the function data 320 is calculated (step S15). Further, the unit period calculation unit 240 stores the calculation result of the combined product as the differential value data 330 in the storage unit 300 (step S16).

Further, the unit cycle calculation unit 240 reads out the differential value data 330 stored in the storage unit 300. Then, when the differential value data 330 is inverted from positive to negative (step S17 = YES), the unit period calculation unit 240 reads out the zero cross point data 340 immediately before being stored in the storage unit 300. Further, the unit cycle calculation unit 240 calculates the difference between the immediately preceding zero cross point data 340 and the current time as the unit cycle data 350, and stores the calculated latest unit cycle data 350 in the storage unit 300 (step). S18). Further, the unit cycle calculation unit 240 updates the unit cycle data 350 stored in the storage unit 300 with the latest unit cycle data 350 in the previous step S18, and then stores the current time in the storage unit 300. It is updated as zero cross point data 340 (step S19).

On the other hand, when the differential value data 330 is not inverted from positive to negative (step S17 = NO), the unit period calculation unit 240 shifts the process to step S20 without going through the processes of steps S18 and S19. ..

Then, when the ratio of the pixels in the face region in which the unit period data 350 stored in the storage unit 300 is different from the initial value is equal to or more than a predetermined ratio (step S20 = YES) , The mode value of the unit cycle data 350 is determined as the pulse cycle (step S21). That is, the statistical processing unit 250 requires that the unit cycle data 350 has already been calculated and the ratio of the pixels whose unit cycle data 350 has been updated from the initial value is equal to or greater than a predetermined ratio among the pixels in the face region. In addition, the unit cycle data 350 of each pixel is statistically processed to determine the pulse cycle.

On the other hand, when the ratio of the pixels in the face region whose unit period data 350 stored in the storage unit 300 is different from the initial value is less than a predetermined ratio (step S20 = NO), the statistical processing unit 250 It is determined that the pulse cycle detection error (step S22).

Further, in the previous step S14, the statistical processing unit 250 also even when the number of data of the time series waveform data 310 stored in the storage unit 300 has not reached the predetermined number of data (step S14 = NO). It is determined that the pulse cycle detection error (step S22).

The above processing is repeatedly executed to continuously detect the pulse cycle. By continuously detecting, the latest pulse cycle can always be detected, but depending on the application, the detection may be performed only once or may be stopped at any time.

When the pulse cycle is determined by the above processing, the time required to start the detection is shorter than the pulse cycle of 3 at the longest. The breakdown is mainly the time to acquire the number of data required to calculate the differential value data 330 as the combined product with the reference function data 320, and the data until the differential value data 330 crosses zero twice. It's time to do it. If there is a time of two pulse cycles at the longest, the differential value data 330 zero-crosses twice, so that all the unit cycle data 350 are updated from the initial values. The time required to acquire the number of data required to calculate the differential value data 330 is about 0.6 seconds when the window width of the function shown in FIG. 3 is "25" and the frame rate of the camera is 40 FPS, and the pulse is high. If it is 80 times / minute, it is less than one cycle of pulse. Therefore, the time required to start the detection is shorter than the pulse of 3 cycles at the longest, and if the pulse is 80 times / minute, the detection is started in about 2.3 seconds at the longest.

Further, the timing of pulse cycle detection is delayed by at least about half the window width of the function shown in FIG. 3 when calculating the differential value data 330. If the window width is "25", a delay of 12 frames will occur. For example, when the frame rate of the camera is 40 FPS, it is 0.3 seconds. By increasing or decreasing the window width of the function, the time required to start detection can be adjusted.

Next, the operation of the pulse cycle detection device 100 of the present embodiment will be described.

Generally, even if the blood flow rate on the face surface changes due to the pulse, the amount of change in the brightness value of each color included in the face region is very small. Therefore, it is not always possible to analyze the pulse period of a person based on the brightness value of each color for all the pixels included in the facial region.

In this regard, in the present embodiment, the unit cycle data 350 is calculated for each pixel based on the brightness value of each color included in the face region, and statistical processing is performed on the calculated unit cycle data 350 for each pixel. By doing so, the representative value of the unit cycle data 350 for each pixel is determined as the pulse cycle of the person. As a result, even if the calculation result of the unit cycle data 350 in some pixels varies, the pulse cycle of the person is detected. As a result, frequency analysis using FFT (Fast Fourier Transform) becomes unnecessary, and the detection time of the pulse cycle is shortened. Then, by detecting the pulse cycle for a short time, for example, it is possible to detect the instantaneous pulse rate (IHR: Instantaneous Heart Rate) used for evaluating the stress, emotion, etc. of a person.

In particular, in the present embodiment, the entire face region is set as the analysis range, and the pulse cycle of the person is performed by performing statistical processing on the unit cycle data 350 calculated for each pixel from the entire face region. Is being detected. As a result, even if there is an influence of the phase difference due to the blood flow velocity between the upper part and the lower part of the face, it is possible to detect the pulse cycle of the person while reducing the influence of the phase difference.

Further, in the present embodiment, the frequency distribution of the appearance frequency of the unit cycle data 350 for each pixel is obtained, and the unit cycle data 350, which is the mode, is determined as the pulse cycle of the person.

Here, as shown in FIG. 6A, it is assumed that the person in the moving image does not wear glasses or a mask, and the brightness value of the entire face region is accurately obtained. In this case, as shown in FIG. 6B, the unit cycle data 350 for each pixel is detected for the entire face region, and the frequency distribution of the appearance frequency of the unit cycle data 350 is obtained. Then, based on the frequency distribution thus obtained, the unit cycle data 350, which is the mode, is suitably detected as the pulse cycle of the person.

On the other hand, as shown in FIG. 7A, the person in the moving image may be wearing a mask, and the brightness value of the person's mouth may not be accurately obtained. In this case, the unit cycle data 350 is not detected from the low-precision luminance value corresponding to the mouth of the person, or even if the unit cycle data 350 is detected, the value is unreliable. .. However, such unreliable unit cycle data 350 is exceptional data in the data group of the unit cycle data 350 for each pixel for the entire face region, and the frequency of appearance of each tends to be low. Therefore, as shown in FIG. 7 (b), the unit cycle data 350, which is the mode of the frequency distribution of the appearance frequency of the unit cycle data 350 obtained from the facial region, is shown in FIG. 6 (b) above. It shows a tendency that is almost the same as the case. Therefore, the unit cycle data 350, which is the mode in the frequency distribution, is suitably detected as the pulse cycle of the person.

Further, as shown in FIG. 8A, the person in the moving image may be wearing eyeglasses, and the brightness value around the eyes of the person may not be accurately obtained. Similarly, in this case, as shown in FIG. 8 (b), the unit cycle data 350, which is the mode of the frequency distribution of the appearance frequency of the unit cycle data 350 obtained from the facial region, is the above-mentioned FIG. 6 (b). ) Shows the same tendency as the case shown in). Therefore, the unit cycle data 350, which is the mode in the frequency distribution, is suitably detected as the pulse cycle of the person.

That is, among the unit cycle data 350 for each pixel obtained from the face image, the unit cycle data 350, which is the mode, is influenced by the attributes of the person in the moving image, such as whether or not glasses or masks are worn. Hateful. Therefore, by selecting the unit cycle data 350, which is the mode, as the statistical processing for the unit cycle data 350 for each pixel, the pulse cycle can be detected from the person placed in various situations, and the pulse can be detected. Increases the robustness of cycle detection.

The brightness value of each pixel is also affected by the ambient light when the moving image is taken. Therefore, the absolute value of the brightness value of each color for each pixel does not always reflect the pulse period of the person.

In this regard, in the present embodiment, the time-series waveform data 310 for the relative ratio Gr of the green luminance value is detected as a pulse wave, and the unit period data 350 is calculated based on the detected pulse wave. Here, even if the ambient light when the moving image is taken changes, the influence on the brightness value of each color is almost the same. Therefore, since the unit cycle data 350 for each pixel is calculated based on the relative ratio Gr of the green luminance value, the unit cycle data 350 is accurately calculated while suppressing the influence of ambient light.

As described above, according to the first embodiment, the effects listed below can be obtained.
(1) The pulse cycle of a person is detected by performing statistical processing on the unit cycle data 350 for each pixel. As a result, even if the calculation result of the unit cycle data 350 in some pixels varies, it is possible to detect the pulse cycle of the person.

(2) The facial area of the person is set as the analysis range of the unit period data 350. As a result, it is possible to reduce the noise included in the calculation result of the unit cycle data 350 for each pixel even if the skin portion of the person is included in the image.

(3) The unit period data 350 for each pixel is calculated based on the relative ratio Gr of the green luminance value. As a result, the unit period data 350 can be calculated while suppressing the influence of ambient light. Therefore, by performing statistical processing on the unit cycle data 350, it is possible to detect the pulse cycle of a person with high accuracy.

(4) The peak point at which the brightness value becomes the maximum value is detected, and the time difference between adjacent peak points is calculated as the unit period data 350. As a result, the unit cycle data 350 for each pixel is calculated when the peak point for one cycle is detected from the time series data of the luminance value. Therefore, by performing statistical processing on the unit cycle data 350, it is possible to detect the pulse cycle of a person in a short time.

(5) The time-series waveform data 310 is converted by the combined product of the time-series waveform data 310 and the reference function data 320 regarding the brightness value for each pixel, and the unit period data 350 is calculated based on the converted time-series waveform data 310. I tried to do it. As a result, the error component of the time series waveform data 310 regarding the luminance value is reduced. Then, the extreme value of the brightness value is obtained based on the time series data in which the error component is reduced. This makes it possible to improve the detection accuracy of the pulse cycle of a person.

(6) The unit period data 350 for each pixel is statistically processed in the entire facial area that is the analysis range. This makes it possible to suppress the influence of the phase difference due to the blood flow velocity for each pixel and improve the detection accuracy of the pulse cycle of the person.

(7) The mode value of the unit cycle data 350 for each pixel is determined as the pulse cycle of a person. As a result, the pulse cycle of the person is detected based on the unit cycle data 350 for each pixel regardless of the attributes of the person in the image, such as whether or not glasses or masks are worn, so that the detection of the pulse cycle is robust. Can be enhanced.

(8) The pulse rate of a person is determined based on the pulse cycle determined by the statistical processing unit 250. Therefore, by detecting the pulse cycle for a short time, it is possible to detect the instantaneous pulse rate used for evaluating the stress or emotion of a person, for example.

(Second Embodiment)
Next, a second embodiment of the pulse cycle detection device will be described with reference to the drawings. In the second embodiment, the method of calculating the unit cycle data from the face region of the person is different from that of the first embodiment. Therefore, in the following description, a configuration different from the first embodiment will be mainly described, and a duplicate description will be omitted for the same or equivalent configuration as the first embodiment.

In the pulse cycle detection device 100 of the present embodiment, the face detection unit 210 detects wearing objects such as glasses and masks from the face region of the person, and specifies the presence or absence of the wearing items as an attribute of the person. Then, the brightness value acquisition unit 220 sets a region excluding the wearing object from the face region of the person as the analysis range of the unit period data 350, based on the attribute of the person specified by the face detection unit 210.

Here, as shown in FIG. 9A, it is assumed that the person in the moving image does not wear glasses or a mask, and the brightness value of the entire face region is accurately obtained. In this case, of the facial region, the region near the nose is set as the analysis range of the unit period data 350. Then, as shown in FIG. 9B, the frequency distribution of the appearance frequency of the unit cycle data 350 is obtained based on the unit cycle data 350 for each pixel obtained from the analysis range, and based on this frequency distribution. The unit cycle data 350, which is the mode, is suitably detected as the pulse cycle of a person.

On the other hand, as shown in FIG. 10A, the person in the moving image may be wearing a mask, and the brightness value of the person's mouth may not be accurately obtained. In this case, the unit cycle data 350 is not detected from the low-precision luminance value corresponding to the mouth of the person, or even if the unit cycle data 350 is detected, the value is unreliable. .. Therefore, of the face area, the area near the forehead excluding the area corresponding to the mouth of the person is set as the analysis range of the unit cycle data 350, and the unit cycle data 350 obtained from the analysis range becomes highly reliable data. .. In this case, as shown in FIG. 10 (b), the frequency distribution of the appearance frequency of the unit period data 350 is the data with few error components as in the case shown in FIG. 9 (b) above. Then, based on the frequency distribution thus obtained, the unit cycle data 350, which is the mode, is suitably detected as the pulse cycle of the person.

Further, as shown in FIG. 11A, the person in the image may be wearing eyeglasses, and the brightness value around the eyes of the person may not be accurately obtained. In this case, the unit cycle data 350 is not detected from the low-precision luminance value corresponding to the eyes of the person, or even if the unit cycle data 350 is detected, the value is unreliable. .. Therefore, among the facial regions, the region near the chin excluding the region corresponding to the eyes of the person is set as the analysis range of the unit cycle data 350, and the unit cycle data 350 obtained from the analysis range becomes highly reliable data. .. In this case, as shown in FIG. 11B, the frequency distribution of the appearance frequency of the unit period data 350 is the data having few error components as in the case shown in FIG. 9B. Then, based on the frequency distribution thus obtained, the unit cycle data 350, which is the mode, is suitably detected as the pulse cycle of the person.

As described above, according to the second embodiment, in addition to the effects (1) to (8) of the first embodiment, the effects listed below can be obtained.
(9) The analysis range of the unit cycle data 350 is set based on the attributes of the person in the moving image. As a result, the analysis range of the unit cycle data 350 is set in a form suitable for the type of wearable object worn by the person in the image, such as glasses and masks. Therefore, regardless of whether or not the person in the image is wearing the wearable object, the entire data group of the unit cycle data 350 obtained from the analysis range is highly reliable. This makes it possible to improve the detection accuracy of the pulse cycle of a person using the unit cycle data 350 for each pixel.

(Other embodiments)
In addition, each of the above-described embodiments can also be implemented in the following embodiments.
-In each of the above embodiments, the peak point at which the brightness value becomes the maximum value is detected from the time-series waveform data 310 regarding the brightness value, and the time difference between adjacent peak points is detected as the unit period data 350. Alternatively, the peak point at which the brightness value becomes the minimum value may be detected from the time-series waveform data 310 relating to the brightness value, and the time difference between adjacent peak points may be detected as the unit period data 350.

-In each of the above embodiments, the mode of the unit cycle data 350 for each pixel is determined as the pulse cycle of the person. Instead of this, the average value or the median value of the unit cycle data 350 for each pixel may be determined as the pulse cycle of the person. In addition, unreliable data located near the upper limit and the lower limit of the distribution are excluded from the frequency distribution of the frequency of appearance of the unit cycle data 350 for each pixel, and the average value or median value of the remaining data group is set as the person. It may be determined as the pulse cycle of.

-In the second embodiment, the analysis range of the unit cycle data 350 is set based on the attributes of the person specified from the moving image, such as whether or not glasses or masks are worn. Instead of this, the analysis range of the unit cycle data 350 may be set in consideration of the installation environment of the camera 10, such as the orientation of the face of a person photographed by the camera 10.

-In each of the above embodiments, the pulse cycle of the person is selected from the unit cycle data 350 included in the frequency distribution. That is, the resolution of the unit cycle data 350 and the resolution of the pulse cycle of the person were the same. Instead, as shown in FIG. 12, fitting is performed by a function assuming that the frequency distribution of the appearance frequency of the unit period data 350 is a normal distribution, and the pulse period of the person is detected based on the fitted function. You may do so. In the example shown in the figure, fitting is performed by the Gaussian function shown in the following [Equation 2]. In this case, for example, the median value μ of the Gaussian function may be determined as the pulse period of the person.

-In each of the above embodiments, the frequency distribution of the appearance frequency of the unit cycle data 350 is updated by using the unit cycle corresponding to each pixel included in the face region. Instead of this, as shown in FIG. 13, the waveform of the pulse wave based on the brightness value of each pixel may be analyzed, and whether or not the frequency distribution can be updated may be determined based on the analysis result. In the example shown in the figure, frequency analysis using FFT (Fast Fourier Transform) is performed on the pulse wave of a person. In this case, the reliability of the unit period data 350 obtained from the pixels is evaluated based on the frequency components obtained by the frequency analysis. For example, when the frequency component obtained by frequency analysis contains a predetermined amount or more of a frequency component higher than the pulse frequency (corresponding to the reciprocal of the pulse period) obtained from the pulse wave waveform, the unit period data obtained from this pixel is obtained. The reliability of 350 is underestimated. Then, for example, the addition value of the appearance frequency of the unit cycle data 350 in the frequency distribution is lowered, and the weighting of the unit cycle data 350 obtained at the present time from this pixel is lowered to update the frequency distribution of the appearance frequency of the unit cycle data 350. do.

-In each of the above embodiments, the case where the first-order differential Gaussian function is applied as the reference function data 320 used for the combined product with the time-series waveform data 310 regarding the luminance value for each pixel has been described as an example. However, as the reference function data 320, a Gaussian function, a sine function, or a cos function may be applied. For example, when a Gaussian function is used as the reference function data 320, the time-series waveform data 310 is smoothed by the combined product of the time-series waveform data 310 and the Gaussian function, and then the differential processing is performed to obtain the brightness value. The maximum value or the minimum value may be obtained. Further, the time average of the time-series waveform data 310 may be calculated, the time-series waveform data 310 may be smoothed, and then the differential processing may be performed to obtain the maximum value or the minimum value of the brightness value.

-In each of the above embodiments, the unit cycle data 350 for each pixel is calculated based on the relative ratio Gr of the green luminance value to the total luminance value of all colors. However, the luminance value used for calculating the unit period data 350 may be, for example, the relative ratio of the green luminance value to the red luminance value, or the relative ratio of the green luminance value to the blue luminance value. It may be a relative ratio of the green luminance value to the sum of the red and blue luminance values. Further, the pulse cycle may be analyzed by directly using the green luminance value. Further, the pulse cycle may be analyzed by directly using the red luminance value which has a negative correlation with the green luminance value instead of the green luminance value.

-In each of the above embodiments, the pulse period is analyzed for each pixel of the face region of the person. However, the unit area for analyzing the pulse period does not necessarily have to be one pixel, and the unit area may be composed of a plurality of pixels.

-In each of the above embodiments, the case where the pulse cycle is detected in the facial region of a person has been described as an example. However, as a target for detecting the pulse cycle, it is also possible to apply other body parts such as a profile and limbs where the human skin is exposed.

-In each of the above embodiments, a case where the pulse cycle of a person is determined at the time of taking a moving image has been described as an example. Instead of this, a moving image may be taken in advance, and the taken moving image may be read to determine the pulse cycle of the person. In this case, for example, it is possible to search for a scene in which the pulse period changes significantly in a moving image.

-In each of the above embodiments, the time from the peak of the pulse wave to the next peak is calculated as the unit cycle corresponding to the calculation method of RRI (R-R interval), but the reference used for the calculation of the unit cycle. The point is not limited to the peak of the pulse wave. In addition to using the zero crossing point of the first derivative, which is the extreme value of the pulse wave, as the reference point, another reference point may be used, such as using the zero crossing point of the pulse wave as the reference point.

Further, a plurality of reference points may be used properly for each unit area. For example, in addition to the positive to negative zero cross point of the first derivative shown in the above embodiment, the negative to positive zero cross point of the first derivative, and the positive to negative and negative to positive zero cross points of the pulse wave (second order). The unit period may be calculated using a plurality of types of reference points such as (which is also a zero crossing point of differentiation) and the type of reference point first detected for each unit region. The detection of the reference point is accelerated, and the time until the detection is completed can be shortened.

-In each of the above embodiments, the case where the detection of the instantaneous pulse rate through the detection of the pulse cycle for a short time is applied to the evaluation of stress and emotion of a person has been described as an example. However, the application example by detecting the instantaneous pulse rate for a short time is not limited to the field of healthcare, and is applied to, for example, personal authentication of an occupant by an in-vehicle camera or personal authentication of a user in an information processing terminal. Is also possible.

10 ... Camera, 100 ... Pulse cycle detection device, 200 ... Control unit, 210 ... Face detection unit, 220 ... Brightness value acquisition unit, 230 ... Pulse wave detection unit, 240 ... Unit cycle calculation unit, 250 ... Statistical processing unit, 260 ... output unit, 300 ... storage unit, 310 ... time series waveform data, 320 ... reference function data, 330 ... differential value data, 340 ... zero cross point data, 350 ... unit period data.

Claims (12)

It is a pulse cycle detection device that detects the pulse cycle of a person from a time-series image.
A pulse wave detection unit that detects time-series data related to the brightness value for a plurality of unit regions in the image, and a pulse wave detection unit.
Based on the time-series data related to the brightness value, the unit cycle calculation unit that calculates the unit cycle, which is the pulse cycle for each unit region, and the unit cycle calculation unit.
A statistical processing unit that performs statistical processing on a plurality of the unit cycles calculated for different unit regions and determines the pulse cycle of the person.
Equipped with a,
The unit cycle calculation unit monitors the time-series data, and when the latest time-series data reaches the calculation standard of the unit cycle, the previous time when the calculation standard was reached and the calculation standard were reached. The unit cycle is calculated from the difference from the current time,
When the determination value regarding the degree of the unit region for which the unit period has been calculated in the unit region in the image reaches a predetermined value for performing the statistical processing, the statistical processing unit said. A pulse cycle detector that performs statistical processing. The pulse wave detection device according to claim 1, wherein the pulse wave detection unit sets a plurality of the unit regions in the skin region of the person. The pulse cycle detection device according to claim 1 or 2, wherein the pulse wave detection unit detects a relative ratio of brightness values of different light wavelength components as the time series data. The pulse cycle detection device according to any one of claims 1 to 3, wherein the unit cycle calculation unit calculates the latest cycle based on the time series data as the unit cycle. The unit cycle calculation unit detects peak points at which the time series data has a maximum value or a minimum value, and calculates the time difference between adjacent peak points as the unit period according to any one of claims 1 to 4. The pulse cycle detector according to the description. The unit cycle calculation unit converts the time series data by the combined product of the time series data and the reference function, and calculates the unit cycle for each unit area based on the converted time series data. The pulse cycle detection device according to any one of 5 to 5. The pulse according to any one of claims 1 to 6, wherein the statistical processing unit determines the mode, average, or median of a plurality of the unit cycles calculated for different unit regions as the pulse cycle of the person. Period detector. The statistical processing unit assumes that the frequency distribution of the frequency of appearance of the unit cycle in a plurality of the unit regions calculated for different unit regions is a normal distribution, and determines the pulse period of the person based on the assumed normal distribution. The pulse cycle detection device according to any one of claims 1 to 6. The statistical processing unit calculates the reliability of the unit cycle for each unit region based on the time series data, weights the unit cycle for each unit region according to the calculated reliability, and the person. The pulse cycle detection device according to any one of claims 1 to 8, wherein the pulse cycle of the device is determined. The pulse cycle detection device according to any one of claims 1 to 9, wherein the pulse rate is calculated based on the pulse cycle determined by the statistical processing unit. It is a pulse cycle detection method that detects the pulse cycle of a person from a time-series image.
Pulse wave detection processing that detects time-series data related to the brightness value for a plurality of unit regions in the image, and
Based on the time-series data related to the brightness value, the unit cycle calculation process for calculating the unit cycle, which is the pulse cycle for each unit region, and the unit cycle calculation process.
It performs statistical processing on the plurality of the unit periods calculated for different unit areas, seen including a statistical process, the determining the pulse period of the person,
The unit cycle calculation process monitors the time series data, and when the latest time series data reaches the calculation standard of the unit cycle, the previous time when the calculation standard was reached and the calculation standard were reached. The unit cycle is calculated from the difference from the current time,
In the statistical processing for determining the pulse cycle, the determination value regarding the degree of the unit region for which the unit period has been calculated in the unit region in the image reaches a predetermined value for performing the statistical processing. A method for detecting a pulse cycle, which performs statistical processing on the plurality of unit cycles. On the computer
Pulse wave detection processing that detects time-series data related to brightness values for multiple unit areas in a time-series image,
Based on the time-series data related to the brightness value, the unit cycle calculation process for calculating the unit cycle, which is the pulse cycle for each unit region, and the unit cycle calculation process.
Statistical processing to determine the pulse cycle of a person by performing statistical processing on a plurality of the unit cycles calculated for different unit regions, and
A pulse cycle detection program for execution,
In the unit cycle calculation process, the time-series data is monitored, and the latest time-series data is preceded.
When the calculation standard of the unit cycle is reached, the unit cycle is calculated from the difference between the previous time when the calculation standard is reached and the current time when the calculation standard is reached.
In the statistical processing for determining the pulse cycle, the determination value regarding the degree of the unit region for which the unit period has been calculated in the unit region in the image reaches a predetermined value for performing the statistical processing. A pulse cycle detection program that performs statistical processing on the plurality of unit cycles.

Images